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Applicant(s) 
Serial No. 
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Andreas ENGELSBERG 
To Be Assigned 
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METHOD AND DEVICE FOR DETERMINING 
A DISPLACEMENT OF IMAGES IN 
AN IMAGE SEQUENCE 

To Be Assigned 
To Be Assigned 



Assistant Commissioner 

for Patents 
Washington, D.C. 20231 

PRELIMINARY AMENDMENT AND 
37 C.F.R. $ 1.125 SUBSTITUTE SPECIFICATION STATEMENT 

SIR: 

Please amend without prejudice the above-identified application before examination, as 
set forth below. 



IN THE TITLE : 

Please amend without prejudice the title to be: 
-METHOD AND DEVICE FOR DETERMINING A DISPLACEMENT OF IMAGES IN AN 
IMAGE SEQUENCE-. 

IN THE SPECIFICATION AND ABSTRACT : 

In accordance with 37 C.F.R. § 1.121(b)(3), a Substitute Specification (including the 
Abstract, but without claims) accompanies this response. It is respectfully requested that the 
Substitute Specification (including Abstract) be entered to replace the Specification of record. 



IN THE CLAIMS : 

Without prejudice, please cancel original claims 1 to 13, and please add new claims 14 to 
35 as follows: 
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— 14. (New) A method for determining a displacement of images in an image sequence for 
compensating for a camera movement, a plurality of image regions of the images being available 
for determining the displacement, each of the plurality of image regions being provided at a 
preselected position of the images, and each of the plurality of image regions having preselected 
dimensions, in which there are predetermined numbers of pixels in various directions of the 
image, and for separating an image movement mainly caused by a camera movement from an 
additional movement, which is superimposed on the image movement in subregions of an image 
to be corrected, the method comprising: 

determining a probability of the image movement occurring without the additional 
movement at different positions of an image; 

specifying and fixing, for each of the plurality of image regions, a position and 
dimensions as a function of the probability of the image movement occurring inside at least one 
image region of the plurality of image regions without the additional movement; and 

determining the displacement by preferentially using the at least one image region of the 
plurality of image regions. 

15. (New) The method of claim 14, wherein: 

the method includes: 

correcting a first image of the image sequence by determining the displacement 
from one of (i) first image data of the first image of the image sequence and second image 
data of a second image of the image, and (ii) the first image data of the first image of the 
image sequence and starting image data of a starting image sequence; and 

determining a regional displacement of the at least one image region of the 
plurality of the image regions is determined from one of (i) the first image data of the first 
image and the second image inside the at least one image region, and (ii) the first image 
data of the first image and the second starting image inside the at least one image region; 
and 

the step of determining the displacement uses the regional displacement of the at least one 
image region of the plurality of image regions as the displacement. 

16. (New) The method of claim 14, wherein the position and the dimensions of at least one first 
image region are selected so that the at least one first image region of images to be corrected is 
mainly filled in by an image background. 

2 
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17. (New) The method of claim 14, wherein the position and the dimensions of at least one 
second image region are selected so that the at least one second image region of images to be 
corrected is mainly filled in by an image foreground. 

18. (New) The method of claim 15, wherein at least one of at least one first image region and at 
least one second image region is used as a function of a reliability factor for determining the 
regional displacement to determine the displacement. 

19. (New) The method of claim 15, wherein: 

two first image regions and one second image region are available for image correction; 

and 

the displacement is determined from one of the following, in a descending priority order: 
a mean of regional displacements of the two first image regions, when a reliability 
of the regional displacement determinations for the two first image regions is considered 
sufficient; 

a regional displacement of one of the two first image regions, when a reliability of 
the regional displacement determination is considered sufficient; and 
a regional displacement of the second image region. 

20. (New) The method of claim 19, wherein the method is used for a head-shoulder recording 
situation, the method further comprising: 

selecting the two first image regions to be in a lateral region to the left and right of a 
vertical center line of a predetermined rectangular image; 

selecting a distance of the two first image regions from a bottom image margin to be 
greater than another distance of the two first image regions from a top image margin; and 

selecting the second image region to be near a center of the image. 

21. (New) The method of claim 18, wherein determined regional displacements and reliability 
factors of the determined regional displacements for two image regions are determined by: 

determining a regional displacement and a correlation quotient for each of the two image 
regions; 

determining a threshold value function as a function of the regional displacement of each 
of the two image regions; 

3 



comparing a determined correlation quotient of one of the two image regions to a 
threshold value resulting from the threshold value function for a regional displacement of another 
of the two image regions; and 

determining the reliability factor of the determined regional displacement of each of the 
two image regions to be sufficient when the determined correlation quotient corresponding to the 
one of the two image regions is greater than the threshold value of the threshold value function 
compared to the determined correlation quotient. 

22. (New) The method of claim 21, wherein the determined correlation quotient of one of the 
two image regions is determined by: 

using block-matching to determine displacement correlation values for a plurality of 
possible regional displacements; 

determining a regional displacement having a maximum displacement correlation value to 
be the regional displacement of the image regions; and 

determining the correlation quotient by dividing the maximum displacement correlation 
value by a mean of determined displacement correlation values. 

23. The method of claim 21, wherein: 

the threshold value function assumes a value of a preselected second threshold value by 
for each arbitrary regional displacement less than a preselected first threshold value; 

the threshold value function assumes a value of the preselected second threshold value 
minus a product for each arbitrary regional displacement greater than the preselected first 
threshold value; 

the product includes a preselected slope parameter and a difference as factors; and 
the difference being formed from the arbitrary regional displacement and the preselected 
first threshold value. 

24. (New) The method of claim 18, wherein for each of the image regions, the regional 
displacement is determined using block-displacement information from a block-based encoding 
process by considering the block-displacement information of blocks lying inside a specific 
image region. 
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25. (New) The method of claim 24, wherein for each of the image regions, the regional 
displacement, including a horizontal component and a vertical component, and the reliability 
factor of a determined regional displacement are determined by: 

determining the horizontal component of the regional displacement by generating a first 
frequency distribution for frequencies of different values of the horizontal component of the 
block-displacement information, the horizontal component of the regional displacement 
corresponding to a value of the horizontal component of the block-displacement information, at 
which the first frequency distribution assumes its principal maximum; 

determining the vertical component of the regional displacement by generating a second 
frequency distribution for frequencies of different values of the vertical component of the 
block-displacement information, the vertical component of the regional displacement 
corresponding to a value of the vertical component of the block-displacement information, at 
which the second frequency distribution assumes its principal maximum; 

determining the reliability factor of the determined regional displacement to be sufficient 
when all of the following conditions are satisfied: 

an absolute value of a difference in position of values of the horizontal component 

of the block-displacement information, corresponding to the principal maximum and a 

secondary maximum of the first frequency distribution, is less than a preselected first 

difference threshold; 

the absolute value of a difference in position of values of the vertical component 

of the block-displacement information, corresponding to the principal maximum and a 

secondary maximum of the second frequency distribution, is less than a preselected 

second difference threshold; 

the principal maximum of the first frequency distribution is greater than a first 

frequency threshold; and 

the principal maximum of the second frequency distribution is greater than a 

second frequency threshold. 

26. (New) A device for determining a displacement of images in an image sequence, the device 
comprising: 

a displacement-detection arrangement to perform a process for determining the 
displacement of images in the image sequence for compensating for a camera movement, a 
plurality of image regions of the images being available for determining the displacement, each 
of the plurality of image regions being provided at a preselected position of the images, and each 
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of the plurality of image regions having preselected dimensions, in which there are predetermined 
numbers of pixels in various directions of the image, and for separating an image movement 
mainly caused by a camera movement from an additional movement, which is superimposed on 
the image movement in subregions of an image to be corrected, the process including: 

determining a probability of the image movement occurring without the additional 

movement at different positions of an image; 

specifying and fixing, for each of the plurality of image regions, a position and 

dimensions as a function of the probability of the image movement occurring inside at 

least one image region of the plurality of image regions without the additional movement; 

and 

determining the displacement by preferentially using the at least one image region 
of the plurality of image regions. 

27. (New) The method of claim 15, wherein the second image data of the second image of the 
image sequence precedes directly the first image. 

28. (New) The method of claim 15, wherein the starting image data of the starting image of the 
image sequence precedes directly the first image. 

29. (New) The method of claim 20, wherein the two first image regions are selected to be in the 
lateral region to the left and right of the vertical center line of the predetermined rectangular 
image, and symmetrical to the vertical center line. 

30. (New) The method of claim 20, wherein the second image region is selected to be near the 
center of the image, and symmetric to the vertical center line of the rectangular image. 

3 1 . (New) The method of claim 29, wherein the second image region is selected to be near the 
center of the image, and symmetric to the vertical center line of the rectangular image. 

32. (New) The method of claim 31, wherein the distance of the second image region from the top 
image margin is selected to be greater than the distance of the second image region from the 
bottom image margin. 
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33. (New) The method of claim 20, wherein the distance of the second image region from the top 
image margin is selected to be greater than the distance of the second image region from the 
bottom image margin. 

34. (New) The method of claim 21, wherein the two image regions include the two first image 
regions. 

35. (New) The method of claim 24, wherein the block-displacement information includes 
displacement vectors.—. 

Remarks 

This Preliminary Amendment cancels without prejudice original claims 1 to 13 in 
the underlying PCT Application No. PCT/DEOO/00134, and adds without prejudice new claims 
14 to 35. The new claims conform the claims to U.S. Patent and Trademark Office rules and do 
not add new matter to the application. 

In accordance with 37 C.F.R. § 1.121(b)(3), the Substitute Specification 
(including the Abstract, but without the claims) contains no new matter. The amendments 
reflected in the Substitute Specification (including Abstract) are to conform the Specification and 
Abstract to U.S. Patent and Trademark Office rules or to correct informalities. As required by 37 
C.F.R. § 1.121(b)(3)(iii) and § 1.125(b)(2), a Marked Up Version Of The Substitute Specification 
comparing the Specification of record and the Substitute Specification also accompanies this 
Preliminary Amendment. In the Marked Up Version, underlining indicates added text and 
bracketing indicated deleted text. Approval and entry of the Substitute Specification (including 
Abstract) is respectfully requested. 

The underlying PCT Application No. PCT/DEOO/00134 includes an International 
Search Report, dated June 20, 2000. The Search Report includes a list of documents that were 
uncovered in the underlying PCT Application. A copy of the Search Report accompanies this 
Preliminary Amendment. 

The underlying PCT application also includes an International Preliminary 
Examination Report, dated June 26, 2001. An English translation of the International 
Preliminary Examination Report accompanies this Preliminary Amendment. 
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Applicant asserts that the subject matter of the present application is new, non- 
obvious, and useful. Prompt consideration and allowance of the application are respectfully 
requested. 



Respectfully Submitted, 




lubmitted, A / 



i chard L. Mayer 
(Reg. No. 22,490) 

One Broadway 

New York, NY 10004 

(212) 425-7200 



CUSTOMER NO. 26646 
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METHOD AND DEVICE FOR DETERMINING A 
DISPLACEMENT OF IMAGES IN AN IMAGE SEQUENCE 



FIELD OF THE INVENTION 

The present invention relates to a device and method for 
determining the displacement of images in an image sequence. 

5 BACKGROUND INFORMATION 

An image- stabilization system, which is supposed to 
substantially stabilize an image while recording with a 
camcorder in one's hand, is discussed in the article of H. 
Rindtorff, "Bildstabilisation in Consumer- Camcordern, Funktion 
10 und Wirkungsweise" ("Image Stabilization in Consumer 

Camcorders - Function and Mode of Operation") Fernseh- und 
Kinotechnik (Television and Cinema Technology) , 4 9th year, no. 
1995. 

15 In this context, and as characterized, the image is split up 

into four sectors, in which movement vectors describing the 
displacement of the image are determined. A total movement 
vector, which ideally represents the displacement of the 
camera, is determined from the movement vectors of the 

2 0 individual sectors. 



SUMMARY OF THE INVENTION 

In contrast, it is believed that the exemplary method of the 
present invention has the advantage that image regions, which 

25 are specially used in the sections of an image to determine 
the displacement of the images, are distinguished by as high 
probability as possible, of the image movement occurring 
without the additional movement. In this way, it is believed 
that the displacement of images of an image sequence may be 

3 0 determined in a particularly reliable manner. In addition, it 
is believed that this is conducive to reliably determining the 
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displacement of images in an image sequence, when the position 
and the dimensions of the image regions are fixed. 

It is believed that this may especially allow the displacement 
5 to be reliably determined for a special scenario of moving 
picture communication (full-motion video) , using relatively 
little processing time. A first image region, which may be 
preferentially used from the plurality of image regions to 
determine the displacement, may be used alone for determining 
10 the displacement, by selecting its position and its dimensions 
inside the images, so that other image regions do not have to 
be considered, and thus, the processing time for determining 
the displacement of images is decreased. 

15 It is believed that it may be advantageous that a first image 

of the image sequence is corrected by determining the 
displacement, either from image data of the first image and 
image data of a second image of the image sequence that may 
precede the first image directly, or from image data of the 

2 0 first image and image data of a second input image of the 

image sequence that may precede the first image directly. A 
regional displacement of an image region being used as a 
displacement, and the regional displacement of an image region 
of the plurality of the image regions either being determined 

2 5 from the image data of the first image and the second image 

inside the one image region, or from the image data of the 
first image and the second input image inside the one image 
region. In this manner, only the image data of the first and 
the second images inside the image region, which is used for 

3 0 determining the displacement of the image, are needed for 

processing, which means that the processing time may be 
reduced . 

In addition, it is believed that it may be advantageous that 
35 the position and the dimensions of the at least one first 

image region are selected so that the at least one first image 
region of the images to be corrected is mainly filled in by 
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the image background. Since the image background may rarely 
exhibits additional movements from one image to another image, 
between which the displacement should be determined, in which 
case the additional movements are superimposed on the image 
5 movement that is especially caused by the unintentional 

movement of a camera, a first image region of the images to be 
corrected may be advantageously used for determining the 
displacement, when it is filled in by the image background. 

10 Furthermore, it is believed that it may be advantageous that 

the position and the dimensions of at least one second image 
region are selected in such a manner, that the at least one 
second image region of the images to be corrected is mainly 
filled in by the image foreground. This may allow the 

15 displacement of images of an image sequence to be determined 
in an especially simple manner, and with high accuracy, when, 
for example, the image background is subjected to a sharp, 
additional movement, which is superimposed on the image 
movement that may be caused by the unintentional camera 

2 0 movement. 

To determine the displacement, it is also believed to be 
advantageous to use at least one first image region and/or at 
least one second image region as a function of a margin of 

25 reliability (reliability factor) , for determining the regional 
displacement. This allows the determination of the region 
displacement to be checked, because a measure for the 
reliability of the regional -displacement determination is 
available inside the exemplary method. Therefore, the 

30 determination of a reliability factor for the 

regional -displacement determination is primarily used to 
simply and reliably separate the image movement from the 
additional movement superimposed on it. 

35 Furthermore, it is believed that it may be advantageous that 
two first image regions and an individual second image region 
are available for image correction, the displacement being 
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determined in one of the following three ways, in the order of 
descending priority: 

from the mean of the regional displacements of the two 
5 first image regions, when the reliability factor of the 

regional -displacement determination of the two first 
image sequences is considered to be sufficient; 
from the regional displacement of one of the two first 
image regions, in which the reliability of the 
10 regional -displacement determination is considered to be 

sufficient ; 

from the regional displacement of the second image 
region . 

15 It is believed that this may allow different image regions of 
different priority to be used for determining the displacement 
of images in an image sequence. In particular, image regions, 
which are mainly filled in by the image background, may be 
preferentially used for determining the displacement of 

20 images. The use of the mean of the regional displacements of 

the two first image regions resulting in a reliable 
determination of the displacement. 

The second priority for determining the displacement, using 
25 the regional displacement calculated from the two first image 
regions, is especially selected to minimize the influence of 
objects moving in the background, since an object moving into 
one of the two first image regions leads to the regional 
displacement of the other of the two image regions being used 
3 0 for determining the displacement of images in an image 
sequence . 

On the third level of priority, the displacement of images 
from the regional displacement of the second image region, 
35 especially from the regional displacement of an image region 
that is mainly filled in by the image foreground, is used. 
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In addition, it is believed that it may be advantageous that 
the exemplary method is used for a head- shoulder recording 
situation, the two first image regions being selected to be in 
a lateral image region, to the left and right of, and which 
5 may be symmetrical to, the vertical center line of a 

predetermined rectangular image, the distance of the two first 
image regions from the bottom image margin being selected to 
be greater than the distance of the two first image regions 
from the top image margin, the second image region being 

10 selected to be near the center of the image, which may be 

symmetric to the vertical center line of the rectangular 
image, and the distance of the second image region from the 
top image margin may be selected to be greater than the 
distance of the second image region from the bottom image 

15 margin. 



In the case of a head- shoulder recording situation, such a 
choice of image regions particularly allows the information 
from the image regions to be sensibly used for determining the 
2 0 regional displacements, and sensibly used to determine the 

displacement of images in an image sequence, by prioritizing 
the use of regional displacements. 



In addition, it is believed that it may be advantageous that, 
25 for two image regions, which may be the two first image 

regions, the regional displacement and the reliability of the 
regional -displacement determination are determined by an 
exemplary method having the following steps: 

the regional displacement and a correlation quotient are 
3 0 calculated for each of the two image regions ; 

a threshold value function is determined as a function of 

each ascertained regional displacement of the two image 

regions ; 

each correlation quotient of one of the two image regions 
3 5 is compared to the value, which results from a threshold 

value function for the regional displacement of the other 
of the two image regions ; 
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the reliability of the regional -displacement 
determination for each of the two image regions is viewed 
as being sufficient, when the correlation quotient 
corresponding to the one image region is greater than the 
value of the threshold- value function to be compared to 
the correlation quotient. 

In this manner, the reliability factors for the regional 
displacement of the two first image regions are not determined 
independently of each other, but rather the ascertained 
regional displacement of one of the two first image regions 
determines the value, which the threshold- value function 
assumes, and to which the regional displacement of the other 
of the two first image regions is compared. Therefore, the 
regional displacement of one image region and the reliability 
factor of the regional displacement are not determined 
independently of movements, especially regional displacements, 
in other parts of an image. 

Furthermore, it is believed that it may be advantageous that 
the correlation quotient of one of the two image regions is 
determined according to an exemplary method having the 
following steps: 

using a block-matching method, displacement-correlation 
values are determined for several possible regional 
displacements ; 

the regional displacement, for which the 
displacement-correlation values are maximized, is 
regarded as the regional displacement of the image 
region; 

the correlation quotient is formed by dividing the 
maximum displacement -correlation value by the average 
value of the calculated displacement - correlation values. 

In this manner, the regional displacement is first of all 
determined using the entire image region, since the 
displacement -correlation values are calculated by forming the 
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sum over (of) all the image points of the image region in 
consideration. Secondly, the correlation quotient is 
normalized, since the maximum displacement-correlation value, 
from which the regional displacement results, is divided by 
5 the mean of the calculated displacement -correlation values. 

In addition, it is believed that it may be advantageous that, 
for each arbitrary regional displacement less than a 
preselected first threshold value, the value of a preselected 

10 second threshold value is assumed by the threshold-value 

function, and that, for each arbitrary regional displacement 
greater than the preselected first threshold value, the value 
of the preselected second threshold value minus a product is 
assumed by the threshold-value function. The product 

15 including a preselected slope parameter and a difference as 
factors, and the difference being formed from the arbitrary 
regional displacement and the preselected first threshold 
value . 

20 It is believed that an advantage of this may be that, for 

larger, ascertained regional displacements of an image region, 
the regional displacement is allowed a sufficient margin of 
reliability, even when the correlation coefficient is smaller, 
which, in the case of a small, ascertained regional 

25 displacement, would indicate a correlation that is not as 

good. Due to the dependence of the two first image regions in 
the calculation of the reliability factor for the 
determination of the regional displacement, the result of the 
structure of the threshold- value function is that, in response 

3 0 to a relatively large regional displacement of one of the two 
first image regions, the required reliability of the 
regional -displacement determination, that is, the required 
magnitude of the correlation coefficient, may not be as high. 

35 In another exemplary method for determining a displacement of 
images in an image sequence, block-displacement information 
from a block-based encoding method is used for determining the 
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regional displacement. The block-displacement information, 
which may be displacement vectors, of blocks lying inside the 
specific image region being considered for each of the image 
regions. It is believed that this should result in an 
especially simple and cost-effective implementation of the 
exemplary method according to the present invention, 
especially when, for example, block-displacement information, 
in particular block-displacement vectors, of small image 
blocks can be obtained without additional expenditure, or with 
only a small amount of additional expenditure. 

In addition, an exemplary method according to the present 
invention implements the following steps for each of the image 
regions, to determine the regional displacement that has a 
horizontal and vertical component, and to calculate the 
reliability factor of the regional -displacement determination. 
To calculate the horizontal component of the regional 
displacement, a first frequency distribution is generated 
for the frequencies of different values of the horizontal 
components of the block-displacement information, the 
horizontal component of the regional displacement 
corresponding to the value of the horizontal component of 
the block-displacement information, at which the first 
frequency distribution assumes its principal (global, 
absolute) maximum; 

to calculate the vertical component of the regional 
displacement, a second frequency distribution is 
generated for the frequencies of different values of the 
vertical components of the block-displacement 
information, the vertical component of the regional 
displacement corresponding to the value of the vertical 
component of the block-displacement information, at which 
the second frequency distribution assumes its principal 
maximum; 

the reliability of the regional -displacement 
determination is considered to be sufficient, when all of 
the following conditions are fulfilled; 
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the absolute value of the difference in the position 
of the values of the horizontal component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary (local, 
5 relative) maximum of the first frequency- 

distribution, is less than a preselected, first 
difference threshold; 

the absolute value of the difference in the position 
of the values of the vertical component of the 

10 block-displacement information, which correspond to 

the principal maximum and the secondary maximum of 
the second frequency distribution, is less than a 
preselected, second difference threshold; 
the principal maximum of the first frequency 

15 distribution is greater than a first frequency 

threshold; and 

the principal maximum of the second frequency 
distribution is greater than a second frequency 
threshold . 

20 

By considering the frequencies of the occurring, horizontal 
and vertical components of the already present 
block-displacement information, firstly, the regional 
displacement of an image region may be determined, and 
25 secondly, the reliability factor of the regional -displacement 
determination may be calculated. 

The exemplary device of the present invention, which is for 
determining a displacement of images in an image sequence, is 

3 0 believed to have the advantage that the displacement can 

especially be determined in a quicker and simpler manner by 
implementing the exemplary method steps of the exemplary 
method according to the present invention, in a microchip or 
on a printed circuit board, which should result in the 

3 5 exemplary method of the present invention being carried out in 
a quicker and more cost-effective manner. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows a block diagram of the exemplary device 
according to the present invention. 

5 Figure 2 shows the principal of image stabilization, by- 
determining a displacement of images in an image sequence. 

Figure 3 shows a flowchart for image stabilization. 

10 Figure 4 shows an example of selecting image regions inside an 

image, to implement the exemplary method according to the 
present invention. 

DETAILED DESCRIPTION 

15 Figure 1 shows a block diagram of the exemplary device 
according to the present invention, for determining a 
displacement of images in an image sequence. The exemplary 
device includes an input 10, an output 20, a 
displacement -detection circuit 100, and a magnification 

20 circuit 200. Displacement-detection circuit 100 includes a 

regional -displacement detector 110, an image-storage device 
(display store) 120, and a microcomputer 130. In addition, 
displacement -detection circuit 100 includes an input, which is 
not shown and is connected to input 10 of the exemplary device 

25 according to the present invention, to regional -displacement 
detector 110, and to image-storage device 120. 

In addition, displacement-detection circuit 100 includes an 
output, which is not shown and is connected to microcomputer 

30 130. Regional -displacement detector 110 is also connected to 
microcomputer 130. Magnification circuit 200 includes two 
inputs not shown and an output, which is connected to output 
20 of the exemplary device. The two inputs of magnification 
circuit 200 are each connected to a magnification processor 

35 210, one of the two inputs of magnification circuit 200 being 
connected to input 10 of the exemplary device, and the other 
of the two inputs of magnification circuit 200 being connected 
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to the output of displacement -detection circuit 100. 

Figure 2 shows an example of image stabilization, for example, 
compensating for camera movements. In this context, a first 
5 input image 13 is corrected to form a first output image 23, 

use being made of image information regarding a second input 
image 11, which may precede the first input image in time. 

Second input image 11 includes an image detail, which is 
10 enlarged by magnification to form second output image 21. In 

this context, second input image 11 is completely recorded by 
the camera, but a user of the exemplary device may only see 
the image detail in magnified form, as second output image 21. 
In this case, the image detail is referred to as second image 
15 12, or also as corrected second image 12. 

According to another exemplary method of the present 
invention, corrected second image 12 is used to correct first 
input image 13 . First input image 13 also includes an image 

20 detail, which is referred to here as uncorrected first image 

14 . A displacement 15 can be determined by comparing 
uncorrected first image 14 to second image 12, that is, to 
corrected second image 12, so that uncorrected first image 14 
can be converted into a corrected first image 16, using 

25 displacement 15. In particular, not all of the image data are 
utilized in comparing uncorrected first image 14 to second 
image 12, but rather, just the image data from image regions 
(not shown) of first image 14 and second image 12 . 

3 0 According to another exemplary method of the present 

invention, second image 11 is used to correct first input 
image 13 . Displacement 15 can be determined by comparing first 
input image 13 to second input image 11, so that uncorrected 
first image 14 can be converted into corrected first image 16, 

35 using displacement 15. In particular, not all of the image 

data are utilized in comparing first input image 13 to second 
input image 11, but rather, just the image data from image 
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regions (not shown) of first input image 13 and second input 
image 11. 

Corrected first image 16 can now be shown to the user as a 
5 first output image 23, in accordance with both the first and 

second exemplary methods. Compared to the second output image, 
the displacement of first input image 13 or corrected first 
image 16 can no longer be observed in first output image 23 . 

10 The method steps of the exemplary method according to the 

present invention are shown in Figure 3 by a flow chart, using 
the correction of first input image 13 as an example. The 
regional displacements of image regions are determined in a 
first sequence step 30. In this connection, first input image 

15 13 is compared to corrected second image 12, which especially 
precedes it in time, or the first input image is compared to 
second input image 11, which especially precedes it in time. 
In sequence step 40, the displacement of first image 14 is 
subsequently determined in accordance with the exemplary 

20 method. In third sequence step 50, uncorrected first image 14 
is displaced by displacement vector 15. The result of this 
operation is corrected first image 16. Subsequently, corrected 
first image 16 is enlarged in fourth sequence step 60, which 
results in first output image 23. 

25 

To correct a further input image, the result of third sequence 
step 50 can be made available to first sequence step 30 by 
storing it, which may be in image -storage device 12 0. The 
result of third sequence step 50 is corrected first image 16, 
3 0 which therefore appears in the place of corrected second image 
12, and, in connection with the correction of the further 
input image, is used for determining regional displacements in 
first sequence step 30. 

35 As an alternative, first input image 13 or uncorrected first 
image 14 can, together with calculated displacement 15, also 
be saved in image-storage device 120, to correct an additional 
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input image . 

Figure 4 shows an example of a distribution of image details 
6, 7, 8 inside first input image 14. In this context, two 
5 first image regions 6, 7 are selected to be symmetric to the 

vertical center line of given, rectangular first image 14, 
especially for using the exemplary method of the present 
invention in a head-shoulder recording situation. In this 
context, the distance of the two first image regions 6, 7 from 

10 the bottom image margin is greater than the distance of the 
two first image regions 6, 7 from the top image margin. 
A second image region 8 is selected to be near the center of 
first image 14, which may be symmetric to the vertical center 
line of the rectangular image, the distance of second image 

15 region 8 from the top image margin which may be selected to be 
greater than the distance of the second image region from the 
bottom image margin. 

The exemplary method of the present invention for determining 

2 0 a displacement 15 of images in an image sequence may be used 

for image stabilization, to compensate for camera movements in 
consumer video cameras (camcorders). The digital, full-motion 
image stabilization of the exemplary method renders it 
particularly suitable for mobile moving -picture communication 
25 (mobile full-motion video) devices. Movements caused by the 

mobile use of a full -mot ion video device should be reduced or 
eliminated to the greatest possible extent. 

One aspect of the exemplary method is to derive (deduce) the 

3 0 camera movement from the relative displacement of consecutive 

images, and to extract the subregion, for example, corrected 
first image 16, from an input image, for example, first input 
image 13, the subregion compensating for the camera movement, 
based on the ascertained displacement 15, in particular of 
35 corrected first image 16, with respect to uncorrected first 
image 14 . 
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In the exemplary method, a plurality of image regions 6, 7, 8 
are made available for determining displacement 15. The image 
regions can be clearly (uniquely) determined inside the image, 
using their positions and their dimensions. By advantageously 
5 selecting the position and the dimensions of image regions 6, 

7, 8, an image movement that is mainly generated by a camera 
movement can now be separated in a particularly simple manner, 
from an additional movement, which is superimposed on the 
image movement in subregions of the image to be corrected. 

10 

In addition, the probability of the image movement occurring 
at various positions of the images of an image sequence 
without the additional movement, is determined. The result of 
this is preferential portions (sections) inside the image, 
15 which can be used for separating the image movement from the 
additional movement. For example, the selection of image 
regions 6, 7, 8 represented in Figure 4 is especially 
advantageous for the special recording situation of a speaker 
who is in the center of the image. 

20 

The special characteristics of the recording situation are 
used as a priori knowledge in selecting and fixing image 
regions 6, 7, 8. In this recording situation, it can be 
assumed that the two first image regions 6, 7 are essentially 

2 5 in the background of the image and second image region 8 is 

essentially in the foreground. 

This means that the two first image regions 6, 7 are mainly 
filled up with image data from the image background, and that 

3 0 second image region 8 is mainly filled up with image data from 

the image foreground. This allows the two first image regions 
6, 7 to be advantageously prioritized, so that the 
determination of a displacement 15 of images in an image 
sequence may be derived from the determination of the regional 
35 displacement of the two first image regions 6, 7. Second image 
region 8 is only used for determining displacement 15 of 
images, when the use of regional displacements from the two 
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first image regions 6, 7 merely permits a regional 
displacement, which is encumbered by a large lack of 
reliability or an insufficiently large reliability factor. 

5 In the specified example, this prioritization is synonymous to 

the image stabilization being carried out or performed using 
image -background information. However, this method does not 
exclusively relate to the differentiation and variable 
prioritization of background and foreground information, to 
10 determine a displacement of images in an image sequence, but 

also relates, for example, to the use of criteria such as edge 
detection, absence of edge detection, or similar. 

In the specified example in QCIF format (Quarter Common 
15 Intermediate Format), the two first image regions 6, 7 

typically extend 120 pixels in the vertical direction and 40 
pixels in the horizontal direction. In this image format, 
second image region 8 typically extends over 135 pixels in the 
vertical direction and 85 pixels in the horizontal direction. 

20 

In the exemplary method for determining a displacement 15 of 
images in an image sequence, in particular, for image 
stabilization using a plurality of image regions 6, 7, 8 that 
each essentially cover the foreground or the background, the 
25 most favorable combination of information may be ascertained 

in each case from the subregions, to minimize the influence of 
moving objects in the background. With the aid of a decision 
criterion, a moving object is detected in the image regions, 
which are, for the most part, covered by the background. 

30 

If the two second image regions 6, 7 do not contain a moving 
object, and therefore, do not contain an additional movement 
by the moving object, which is superimposed on the image 
movement and is especially caused by a camera movement, then 
3 5 the displacement to be calculated is determined from the mean 
of the calculated regional displacements of the two first 
image regions 6, 7. If one first image region 6, 7 contains a 
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moving object, then displacement 15 to be determined is 
ascertained from the other image region not having a moving 
object. If both first image regions 6, 7 contain a moving 
object, then the information from second image region 8, the 
5 foreground subregion, is used for determining the movement. 

To determine the regional displacement of an image region 6, 
7, 8, two alternative methods may be used, depending on 
whether or not block-displacement information from a 
10 block-based encoding method, for example, from a block-based 

transmission mode (method) for reducing bandwidth, is 
accessible in a relatively simple way. 

If block-displacement information is not easily accessible 
15 (readily available) , then a block-matching method, by which an 

additional movement, that is, a local movement inside one of 
the two first image regions 6, 7, can be detected, is used for 
determining the regional displacement. A local movement, for 
example, an appearing object, occurring in an image region 6, 
2 0 7, 8 can be detected by evaluating the 

displacement-correlation values from the block-matching 
method . 

To this end, the ratio of the average value of the 
25 displacement-correlation values to the maximum correlation 

value is compared to an adaptive threshold-value function. To 
determine the mean of the displacement -correlation values, the 
sum of all of the determined correlation values is calculated 
and subsequently divided by the number of correlation values. 
30 The maximum displacement-correlation value is assumed for a 
particular displacement. The regional displacement 
corresponding to the maximum displacement -correlation value is 
assumed to be the regional displacement. The correlation 
quotient corresponds to the maximum displacement - correlation 
35 value divided by the mean of the displacement-correlation 
values, and is therefore normalized. 
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An additional movement, that is, a local movement inside the 
image region, is then detected when the correlation quotient 
is less than the value of an adaptive threshold-value 
function. The adapted threshold- value function is a function 
of the length of a displacement vector, which indicates a 
regional displacement. To detect an additional movement or a 
local movement in one of the two first image regions 6, 7, the 
correlation quotient of the first region 6, 7 in consideration 
may be compared to the value of the adaptive threshold-value 
function, the value of the adaptive threshold-value function 
resulting from the vector length of displacement 15 for the 
other of the two first image regions 6, 7. In this context, 
the threshold-value function may be defined as follows: 

for any arbitrary length of the regional -displacement 
vector, that is, for any arbitrary displacement, which is 
less than a preselected, first threshold value, the value 
of a preselected, second threshold value is assumed by 
the threshold-value function; 

for any arbitrary length of the regional -displacement 
vector, which is greater than the preselected, first 
threshold value, the value of the preselected, second 
threshold value minus a product is assumed by the 
threshold value function, the product including a 
preselected slope parameter and a difference as factors, 
and the difference being formed from the arbitrary 
regional displacement and the preselected first threshold 
value . 



The second exemplary method for determining the displacement 
3 0 of images in an image sequence is based on the use of 

block-displacement information from a block-based encoding 
method. As an alternative to calculating displacement, using a 
block-matching method for an image region 6, 7, 8, the 
displacement vectors of small blocks, for example, of size 8x8 
35 or 16x16 can also be used for determining the regional 
displacement of image regions 6, 7, 8, and therefore, 
information from an encoding method can be directly used to 
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reduce the computational time. 

This approach may be particularly of interest if, for example, 
hardware support allows the block-displacement vectors to be 
5 obtained without additional expenditure, or with only a small 
amount of additional expenditure. A local movement inside an 
image region 6, 7, 8, in particular, inside one of the two 
first image regions 6, 7, can be detected in an especially 
simple manner, when displacement vectors of small blocks of 
10 the image are, for example, known from an encoding method. 

In this context, all of the displacement vectors for blocks 
lying within one of image regions 6, 7, 8 are initially 
assigned to the corresponding image region 6, 7, 8. Separate 

15 frequency distributions or histograms are generated for the 
horizontal and vertical components of each of the assigned 
displacement vectors. The result for each image region 6, 7, 8 
is a first frequency distribution for the horizontal component 
and a second frequency distribution for the vertical component 

2 0 of the displacement vectors of the image blocks. 

An additional movement or a local movement is detected by 
analyzing the frequency distributions assigned to the one 
image region. A local movement is detected, if the difference 
25 of the positions between the principal maximum and the 
greatest secondary maximum of one , of the two assigned 
frequency distributions exceeds a certain threshold value, and 
the magnitude of the principal maxima falls below a threshold 
value . 

30 

The determination of a local or additional movement inside an 
image region means that the regional displacement could not be 
calculated with sufficient reliability. Consequently, the 
calculation of the reliability factor of the 
35 regional -displacement determination yields a negative result 
with regard to the regional -displacement determination. 
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The exemplary method for determining the regional 
displacement, including its horizontal and vertical 
components, and for calculating the reliability of the 
regional-displacement determination, is described as follows: 
to calculate the horizontal component of the regional 
displacement, a first frequency distribution is generated 
for the frequencies of different values of the horizontal 
components of the block-displacement information, the 
horizontal component of the regional displacement 
corresponding to the value of the horizontal component of 
the block-displacement information, at which the first 
frequency distribution assumes its principal maximum; 
to calculate the vertical component of the regional 
displacement, a second frequency distribution is 
generated for the frequencies of different values of the 
vertical components of the block-displacement 
information, the vertical component of the regional 
displacement corresponding to the value of the vertical 
component of the block-displacement information, at which 
the second frequency distribution assumes its principal 
maximum ; 

the reliability of the regional -displacement 
determination is considered to be sufficient, when all of 
the following conditions are fulfilled: 

the absolute value of the difference in the position 
of the values of the horizontal component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
the first frequency distribution, is less than a 
preselected, first difference threshold; 
the absolute value of the difference in the position 
of the values of the vertical component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
the second frequency distribution, is less than a 
preselected, second difference threshold; 
the principal maximum of the first frequency 
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distribution is greater than a first frequency 
threshold; and 

the principal maximum of the second frequency- 
distribution is greater than a second frequency 
threshold; 



As an alternative to the block-matching method for an image 
region 6, 7, 8 in question, block-displacement vectors of a 
block-oriented image encoding method can therefore be used to 

10 determine the displacement of images in an image sequence, to 
reduce the computational time. For the block-matching method, 
a criterion was specified which is suited for detecting local 
movements inside relevant image region 6, 7, 8, and is 
synonymous to the reliability of the regional -displacement 

15 determination not being sufficient. A criterion for detecting 
local movements inside relevant image regions 6, 7, 8 was 
likewise specified for the displacement -estimation method 
using block-displacement vectors. 
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ABSTRACT OF THE DISCLOSURE 
A device and a method for determining a displacement of images 
in an image sequence, which are specially used to compensate 
for a camera movement. A plurality of image regions of the 
5 images is available for determining the displacement, each 

image region being provided at a preselected position of the 
images, each image region having preselected dimensions, in 
particular, a predetermined number of pixels in various 
directions of the image, and an image movement mainly caused 

10 by a camera movement being separated from an additional 
movement that is superimposed on the image movement in a 
subregion of the image to be corrected, by calculating the 
probability of the movement occurring without the additional 
movement is calculated at different positions of an image, 

15 specifying and fixing for each image region, the position and 
the dimensions as a function of the probability of the image 
movement occurring inside the one image region without the 
additional movement, and preferentially using at least one 
image region out of the plurality of image regions for 

2 0 determining the displacement. 

405874 
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MARKED UP VERSION OF SUBSTITUTE SPECIFICATION 

[10191/1974] 

METHOD AND DEVICE FOR DETERMINING A 
DISPLACEMENT OF IMAGES IN AN IMAGE SEQUENCE 



[Background Information 
] FIELD OF THE INVENTION 

The present invention [starts out froml relates to a device and 
method for determining the displacement of images in an image 
5 sequence [ according to the species defined in the main claim, 
and from a device for determining the displacement of images 
in an image sequence according to the species defined in the 
other independent Claim 13. ] ^_ 

10 BACKGROUND INFORMATION 

An image- stabilization system, which is supposed to 
substantially stabilize an image while recording with a 
camcorder in one's hand, is [known froml discussed in the 
article of H. Rindtorff, "Bildstabilisation in 

15 Consumer -Camcordern, Funktion und Wirkungsweise " ("Image 

Stabilization in Consumer Camcorders - Function and Mode of 
Operation") Fernseh- und Kinotechnik (Television and Cinema 
Technology) , 49th year, no. Yt, 1995. [ ] 

2 0 In this context , and as characterized , the image is split up 

into four sectors, in which movement vectors describing the 
displacement of the image are determined. A total movement 
vector, which ideally represents the displacement of the 
camera, is determined from the movement vectors of the 

25 individual sectors. 

[Summary of the Invention 
] SUMMARY OF THE INVENTION 

In contrast, it is believed that the exemplary method of the 
30 present invention [, possessing the features of the main 
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claim,] has the advantage that image regions, which are 
specially [utilized] used in the sections of an image [ in 
order] to determine the displacement of the images, are 
distinguished by as high probability as possible, of the image 
5 movement occurring without the additional movement. [By] In 
this fmeans.l way, it is believed that the displacement of 
images of an image sequence [is] may be determined in a 
particularly reliable manner. In addition, it is believed that 
this is conducive to [the ] reliabl [e determination of]y 
10 determining the displacement of images in an image sequence, 
when the position and the dimensions of the image regions are 
fixed. [T] 

It is believed that t his may especially allow [s] the 
15 displacement to be reliably determined for a special scenario 
of moving picture communication (full -motion video) , using 
relatively little processing time. A first image region, which 
r is] may be preferentially [utilized] used from the plurality of 
image regions to determine the displacement, may be used alone 
20 for determining the displacement, by [ means of] selecting its 
position and its dimensions inside the images, so that other 
image regions do not have to be considered, and thus, the 
processing time for determining the displacement of images is 
decreased. 

25 

[Advantageous further refinements and improvements of the 
method indicated in the main claim are rendered possible by 
measures specified in the dependent claims. 

3 0 It is particularly] It is believed that it may be advantageous 

that a first image of the image sequence is corrected by 
determining the displacement, either from image data of the 
first image and image data of a second image of the image 
sequence that [preferably] may precede [s] the first image 
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directly, or from image data of the first image and image data 
of a second input image of the image sequence that 

[preferably] may precede [s] the first image directly [; a] A 

regional displacement of an image region being [utilized] used 
5 as a displacement, and the regional displacement of an image 
region of the plurality of the image regions either being 
determined from the image data of the first image and the 
second image inside the one image region, or from the image 
data of the first image and the second input image inside the 
10 one image region. In this manner, only the image data of the 
first and the second images inside the image region, which is 
used for determining the displacement of the image, are needed 
for processing, which means that the processing time [is] may 
be reduced. 

15 

In addition, it is believed that it may be advantageous that 
the position and the dimensions of the at least one first 
image region are selected [in such a manner,] so that the at 
least one first image region of the images to be corrected is 

20 mainly filled in by the image background. Since [, as a rule,] 
the image background [ , ] may rarely exhibits additional 
movements from one image to another image, between which the 
displacement should be determined, in which case the 
additional movements are superimposed on the image movement 

25 that is especially caused by the unintentional movement of a 
camera, a first image region of the images to be corrected 
[is] may be advantageously used for determining the 
displacement, when it is filled in by the image background. 

3 0 Furthermore, it is believed that it may be advantageous that 
the position and the dimensions of at least one second image 
region are selected in such a manner, that the at least one 
second image region of the images to be corrected is mainly 
filled in by the image foreground. This may allow [s] the 
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displacement of images of an image sequence to be determined 
in an especially simple manner, and with high accuracy, when, 
re.g.l for example, the image background is subjected to a 
sharp, additional movement, which is superimposed on the image 
5 movement that may be caused by the unintentional camera 
movement . 

[In order t]To determine the displacement, it is also believed 
to be advantageous [that thel to use at least one first image 

10 region and/or [ the] at least one second image region [ is used] 
as a function of a margin of reliability (reliability factor) , 
for determining the regional displacement. This allows the 
determination of the region displacement to be checked, 
because a measure for the reliability of the 

15 regional -displacement determination is available inside the 
exemplary method. Therefore, the determination of a 
reliability factor for the regional -displacement determination 
is primarily used to simply and reliably separate the image 
movement from the additional movement superimposed on it. 

20 

Furthermore, it is believed that it may be advantageous that 
two first image regions and an individual second image region 
are available for image correction, the displacement being 
determined in one of the following three ways, in the order of 
2 5 descending priority: 

from the mean of the regional displacements of the two 
first image regions, when the reliability factor of the 
regional -displacement determination of the two first 
30 image sequences is considered to be sufficient; 

from the regional displacement of one of the two first 
image regions, in which the reliability of the 
regional -displacement determination is considered to be 
sufficient ; 
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from the regional displacement of the second image 
region . 

I"T1 It is believed that t his may allow [s] different image 
5 regions of different priority to be used for determining the 
displacement of images in an image sequence. In particular, 
[it is possible to preferentially utilize ] image regions, 
which are mainly filled in by the image background, may be 
preferentially used for determining the displacement of 

10 images [; t]_. The use of the mean of the regional 

displacements of the two first image regions resulting in a 
reliable determination of the displacement . [ ] 

The second priority for determining the displacement, using 
15 the regional displacement calculated from the two first image 
regions, is especially selected [ in order] to minimize the 
influence of objects moving in the background, since an object 
moving into one of the two first image regions leads to the 
regional displacement of the other of the two image regions 
2 0 being used for determining the displacement of images in an 
image sequence . [ ] 

On the third level of priority, the displacement of images 
from the regional displacement of the second image region, 
25 especially from the regional displacement of an image region 
that is mainly filled in by the image foreground, is used. 

In addition, it is believed that it may be advantageous that 
the exemplary method is used for a head- shoulder recording 
30 situation, the two first image regions being selected to be in 
a lateral image region, to the left and right of, and 
[preferably! which may be symmetrical to, the vertical center 
line of a predetermined rectangular image, the distance of the 
two first image regions from the bottom image margin being 
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selected to be greater than the distance of the two first 
image regions from the top image margin, the second image 
region being selected to be near the center of the image, 
rpreferablvl which may be symmetric to the vertical center line 
5 of the rectangular image, and the distance of the second image 
region from the top image margin [being preferably] may be 
selected to be greater than the distance of the second image 
region from the bottom image margin. [ ] 

10 In the case of a head-shoulder recording situation, such a 
choice of image regions particularly allows the information 
from the image regions to be sensibly used for determining the 
regional displacements, and sensibly used to determine the 
displacement of images in an image sequence, by prioritizing 

15 the [utilization] use of regional displacements. 

In addition, it is believed that it may be advantageous that, 
for two image regions, [preferably] which may be the two first 
image regions, the regional displacement and the reliability 

2 0 of the regional -displacement determination are determined by 

fal an exemplary method having the following steps: 

the regional displacement and a correlation quotient are 
calculated for each of the two image regions ; 
a threshold value function is determined as a function of 
25 each ascertained regional displacement of the two image 

regions ; 

each correlation quotient of one of the two image regions 
is compared to the value, which results from a threshold 
value function for the regional displacement of the other 

3 0 of the two image regions ; 

the reliability of the regional -displacement 
determination for each of the two image regions is viewed 
as being sufficient, when the correlation quotient 
corresponding to the one image region is greater than the 
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value of the threshold-value function to be compared to 
the correlation quotient. 

In this manner, the reliability factors for the regional 
5 displacement of the two first image regions are not determined 
independently of each other, but rather the ascertained 
regional displacement of one of the two first image regions 
determines the value, which the threshold-value function 
assumes, and to which the regional displacement of the other 
10 of the two first image regions is compared- Therefore, the 

regional displacement of one image region and the reliability 
factor of the regional displacement are not determined 
independently of movements, especially regional displacements, 
in other parts of an image. 

15 

Furthermore, it is believed that it may be advantageous that 
the correlation quotient of one of the two image regions is 
determined according to [a] an exemplary method having the 
following steps: 

2 0 - using a block-matching method, displacement -correlation 

values are determined for several possible regional 
displacements; 

the regional displacement, for which the 
displacement -correlation values are maximized, is 
25 regarded as the regional displacement of the image 

region ,- 

the correlation quotient is formed by dividing the 
maximum displacement-correlation value by the average 
value of the calculated displacement-correlation values. 

30 

In this manner, the regional displacement is first of all 
determined using the entire image region, since the 
displacement -correlation values are calculated by forming the 
sum over (of) all the image points of the image region in 
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consideration. Secondly, the correlation quotient is 
normalized, since the maximum displacement-correlation value, 
from which the regional displacement results, is divided by 
the mean of the calculated displacement -correlation values. 

In addition, it is believed that it may be advantageous that, 
for each arbitrary regional displacement less than a 
preselected first threshold value, the value of a preselected 
second threshold value is assumed by the threshold-value 
function, and that, for each arbitrary regional displacement 
greater than the preselected first threshold value, the value 
of the preselected second threshold value minus a product is 

assumed by the threshold-value function [; t]^ The product 

including a preselected slope parameter and a difference as 
factors, and the difference being formed from the arbitrary 
regional displacement and the preselected first threshold 
value . [ The] 

It is believed that an advantage of this [is] may be that, for 
larger, ascertained regional displacements of an image region, 
the regional displacement is allowed a sufficient margin of 
reliability, even when the correlation coefficient is smaller, 
which, in the case of a small, ascertained regional 
displacement, would indicate a correlation that is not as 
good. Due to the dependence of the two first image regions in 
the calculation of the reliability factor for the 
determination of the regional displacement, the result of the 
structure of the threshold-value function is that, in response 
to a relatively large regional displacement of one of the two 
first image regions, the required reliability of the 
regional -displacement determination, that i [ . e . ] s , the 
required magnitude of the correlation coefficient, [are] may 
not be as high. 
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In [a further, specific embodiment of thel another exemplary 
method for determining a displacement of images in an image 
sequence, [ it is advantageous that] block-displacement 
information from a block-based encoding method is used for 

5 determining the regional displacement [ ; t]^ The 

block-displacement information, [preferably! which may be 
displacement vectors, of blocks lying inside the specific 
image region being considered for each of the image regions. 
[Tl It is believed that t his should result [s] in an especially 

10 simple and cost-effective implementation of the exemplary 

method according to the present invention, especially when, 
re.g.l for example, block-displacement information, in 
particular block-displacement vectors, of small image blocks 
can be obtained without additional expenditure, or with only a 

15 small amount of additional expenditure. 

In addition, [it is particularly advantageous for the second 
specific embodiment of thel an exemplary method according to 
the present invention [, that a method having] implements the 

2 0 following steps [ is implemented] for each of the image 

regions, [ in order] to determine the regional displacement 

that has a horizontal and vertical component, and [ in order] 

to calculate the reliability factor of the 

regional -displacement determination. 
25 - [In order t]To calculate the horizontal component of the 

regional displacement, a first frequency distribution is 
generated for the frequencies of different values of the 
horizontal components of the block-displacement 
information, the horizontal component of the regional 

3 0 displacement corresponding to the value of the horizontal 

component of the block-displacement information, at which 
the first frequency distribution assumes its principal 
(global, absolute) maximum; 

[in order ] to calculate the vertical component of the 
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regional displacement, a second frequency distribution is 
generated for the frequencies of different values of the 
vertical components of the block-displacement 
information, the vertical component of the regional 
5 displacement corresponding to the value of the vertical 

component of the block-displacement information, at which 
the second frequency distribution assumes its principal 
maximum; 

the reliability of the regional -displacement 
10 determination is considered to be sufficient, when all of 

the following conditions are fulfilled; 

the absolute value of the difference in the position 
of the values of the horizontal component of the 
block-displacement information, which correspond to 
15 the principal maximum and the secondary (local, 

relative) maximum of the first frequency 
distribution, is less than a preselected, first 
difference threshold; 

the absolute value of the difference in the position 
2 0 of the values of the vertical component of the 

block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
the second frequency distribution, is less than a 
preselected, second difference threshold; 
2 5 - the principal maximum of the first frequency 

distribution is greater than a first frequency 
threshold; and 

the principal maximum of the second frequency 
distribution is greater than a second frequency 
30 threshold. 

[Using simple means, in particular, b] By considering the 
frequencies of the occurring, horizontal and vertical 
components of the already present block-displacement 
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information, [it is thereby possible to, first of 
all] firstly, [ determine] the regional displacement of an image 
region may be determined , and secondly, [ to calculate] the 
reliability factor of the regional -displacement determination 
5 may be calculated . 

The exemplary device of the present invention, which is for 
determining a displacement of images in an image sequence, 
[and possesses the features of the other independent Claim 

10 13.1 is believed to ha[s]ve the advantage that the displacement 
can especially be determined in a quicker and simpler manner 
by implementing the exemplary method steps of the exemplary 
method according to the present invention, in a microchip or 
on a printed circuit board, which should result [s] in the 

15 exemplary method of the present invention being carried out in 
a quicker and more cost-effective manner. 

[Brief Description of the Drawing 

2 0 An exemplary embodiment of the present invention is 

represented in the drawing and explained in detail in the 
following description. 

The figures show: 

25 

Figure 1 1 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows a block diagram of the exemplary device 

according to the present invent ion [ ; ] ^_ 

30 Figure 2 [ ] shows the principal of image stabilization, by 

determining a displacement of images in an 
image sequence [;]_=_ 

Figure 3 [ ] shows a flowchart for image stabilization [ ,- 
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and] ^ 

Figure 4 [ 1 shows an example of selecting image regions 

inside an image, [in order ] to implement the 
5 exemplary method according to the present 

invention . 

[Description of the Exemplary Embodiment 
] DETAILED DESCRIPTION 

10 Figure 1 shows a block diagram of the exemplary device 
according to the present invention, for determining a 
displacement of images in an image sequence. The exemplary 
device [ according to the present invention] includes an input 
10, an output 20, a displacement -detect ion circuit 100, and a 

15 magnification circuit 200. Displacement -detect ion circuit 100 
includes a regional -displacement detector 110, an 
image-storage device (display store) 120, and a microcomputer 
130. In addition, displacement-detection circuit 100 includes 
an input, which is not shown and is connected to input 10 of 

2 0 the exemplary device according to the present invention, to 
regional -displacement detector 110, and to image-storage 
device 12 0 . 

In addition, displacement-detection circuit 100 includes an 
25 output, which is not shown and is connected to microcomputer 

130 [; r]^ Regional -displacement detector 110 is also [ being] 

connected to microcomputer 130. Magnification circuit 200 
includes two inputs not shown and an output, which is 
connected to output 20 of the exemplary device. The two inputs 
30 of magnification circuit 200 are each connected to a 
magnification processor 210, one of the two inputs of 
magnification circuit 200 being connected to input 10 of the 
exemplary device, and the other of the two inputs of 
magnification circuit 200 being connected to the output of 
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displacement-detection circuit 100. 

[A] Figure 2 shows a n example of image stabilization, [e.g.] for 
example , compensating for camera movements [ , is represented in 
5 Figure 2] . In this context, a first input image 13 is 

corrected to form a first output image 23, use being made of 
image information regarding a second input image 11, which 
[preferably] may. precede [s] the first input image in time. 

10 Second input image 11 includes an image detail, which is 

enlarged by magnification to form second output image 21. In 
this context, second input image 11 is completely recorded by 
the camera, but a user of the exemplary device [ according to 
the present invention] may only see the image detail in 

15 magnified form, as second output image 21. In this case, the 
image detail is referred to as second image 12, or also as 
corrected second image 12 . 

According to [a first specific embodiment of the] another 
2 0 exemplary method of the present invention, corrected second 

image 12 is used to correct first input image 13 . First input 
image 13 also includes an image detail, which is referred to 
here as uncorrected first image 14 . A displacement 15 can be 
determined by comparing uncorrected first image 14 to second 

2 5 image 12, that i [ . e . ] s , to corrected second image 12, so that 

uncorrected first image 14 can be converted into a corrected 
first image 16, using displacement 15. In particular, not all 
of the image data are utilized in comparing uncorrected first 
image 14 to second image 12, but rather, just the image data 

3 0 from image regions (not shown) of first image 14 and second 

image 12 . 

According to [a second specific embodiment of the] another 
exemplary method of the present invention, second image 11 is 
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used to correct first input image 13 . Displacement 15 can be 
determined by comparing first input image 13 to second input 
image 11, so that uncorrected first image 14 can be converted 
into corrected first image 16, using displacement 15. In 
5 particular, not all of the image data are utilized in 

comparing first input image 13 to second input image 11, but 
rather, just the image data from image regions (not shown) of 
first input image 13 and second input image 11. 

10 Corrected first image 16 can now be shown to the user as a 
first output image 23, in accordance with both the first 
[specific embodiment ] and [ the] second [specific embodiment of 
the method according to the present invention] exemplary- 
methods . Compared to the second output image, the displacement 

15 of first input image 13 or corrected first image 16 can no 
longer be observed in first output image 23 . 

The [essential ] method steps of the exemplary method according 
to the present invention are [represented] shown in Figure 3 by 

20 a flow chart, using the correction of first input image 13 as 
an example. The regional displacements of image regions are 
determined in a first sequence step 30. In this connection, 
first input image 13 is compared to corrected second image 12, 
which especially precedes it in time, or the first input image 

25 is compared to second input image 11, which especially 

precedes it in time. In sequence step 40, the displacement of 
first image 14 is subsequently determined in accordance with 
the exemplary method [ of the present invention] . In third 
sequence step 50, uncorrected first image 14 is displaced by 

30 displacement vector 15. The result of this operation is 

corrected first image 16. Subsequently, corrected first image 
16 is enlarged in fourth sequence step 60, which results in 
first output image 23 . 
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[In order t]To correct a further input image [ in accordance 
with the method of the present invention] , the result of third 
sequence step 50 can be made available to first sequence step 
3 0 by storing it, [preferably! which may be in image- storage 
5 device 120. The result of third sequence step 50 is corrected 
first image 16, which therefore appears in the place of 
corrected second image 12, and, in connection with the 
correction of the further input image, is used for determining 
regional displacements in first sequence step 30. 

10 

As an alternative, first input image 13 or uncorrected first 
image 14 can, together with calculated displacement 15, also 
be saved in image-storage device 120, [in order ] to correct an 
additional input image. 

15 

[Represented in ] Figure 4 [is] shows an example of a 
distribution of image details 6, 7, 8 inside first input image 
14. In this context, two first image regions 6, 7 are selected 
to be symmetric to the vertical center line of given, 

20 rectangular first image 14, especially for using the exemplary 
method of the present invention in a head- shoulder recording 
situation. In this context, the distance of the two first 
image regions 6, 7 from the bottom image margin is greater 
than the distance of the two first image regions 6, 7 from the 

2 5 top image margin. 

A second image region 8 is selected to be near the center of 
first image 14, [preferably] which may be symmetric to the 
vertical center line of the rectangular image, the distance of 
second image region 8 from the top image margin [being 

30 preferably] which may be selected to be greater than the 

distance of the second image region from the bottom image 
margin . 

The exemplary method of the present invention for determining 



15 



MARKED UP VERSION OF SUBSTITUTE SPECIFICATION 



a displacement 15 of images in an image sequence [can 
preferably] may be used for image stabilization, [ in order] to 
compensate for camera movements in consumer video cameras 
(camcorders). The digital, full-motion image stabilization of 
5 the exemplary method renders it particularly suitable for 

mobile moving-picture communication (mobile full -mot ion video) 
devices. Movements caused by the mobile use of a full -motion 
video device should be reduced or eliminated to the greatest 
possible extent . [ The basic principle] 

10 

One aspect of the exemplary method is to derive (deduce) the 
camera movement from the relative displacement of consecutive 
images, and to extract the subregion, re.g.l for example, 
corrected first image 16, from an input image, [e.g.] for 
15 example , first input image 13, the subregion compensating for 
the camera movement, based on the ascertained displacement 15, 
in particular of corrected first image 16, with respect to 
uncorrected first image 14. 

20 In the exemplary method [ according to the present invention], 

a plurality of image regions 6, 7, 8 are made available for 
determining displacement 15. The image regions can be clearly 
(uniquely) determined inside the image, using their positions 
and their dimensions. By advantageously selecting the position 

2 5 and the dimensions of image regions 6, 7, 8, an image movement 
that is mainly generated by a camera movement can now be 
separated in a particularly simple manner, from an additional 
movement, which is superimposed on the image movement in 
subregions of the image to be corrected. [ ] 

30 

In addition, the probability of the image movement occurring 
at various positions of the images of an image sequence 
without the additional movement, is determined. The result of 
this is preferential portions (sections) inside the image. 
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which can be used for separating the image movement from the 
additional movement. For example, the selection of image 
regions 6, 7, 8 represented in Figure 4 is especially 
advantageous for the special recording situation of a speaker 
5 who is in the center of the image. 

The special characteristics of the recording situation are 
used as a priori knowledge in selecting and fixing image 
regions 6, 7, 8. In this recording situation, it can be 
10 assumed that the two first image regions 6, 7 are essentially 
in the background of the image and second image region 8 is 
essentially in the foreground. 

This means that the two first image regions 6, 7 are mainly 
15 filled up with image data from the image background, and that 
second image region 8 is mainly filled up with image data from 
the image foreground. This allows the two first image regions 

6, 7 to be advantageously prioritized, so that the 
determination of a displacement 15 of images in an image 

20 sequence [is preferably] may be derived from the determination 
of the regional displacement of the two first image regions 6, 

7. Second image region 8 is only used for determining 
displacement 15 of images, when the use of regional 
displacements from the two first image regions 6, 7 merely 

25 permits a regional displacement, which is encumbered by a 
large lack of reliability or an insufficiently large 
reliability factor. 

In the specified example, this prioritization is synonymous to 
30 the image stabilization being [preferentially ] carried out or 
performed using image -background information. However, this 
method does not exclusively relate to the differentiation and 
variable prioritization of background and foreground 
information, [ in order] to determine a displacement of images 
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in an image sequence, but also relates, for example, to the 
use of criteria such as edge detection, absence of edge 
detection, or similar. 

5 In the specified example in QCIF format (Quarter Common 

Intermediate Format), the two first image regions 6, 7 
typically extend 12 0 pixels in the vertical direction and 40 
pixels in the horizontal direction. In this image format, 
second image region 8 typically extends over 13 5 pixels in the 
10 vertical direction and 85 pixels in the horizontal direction. 

In the exemplary method for determining a displacement 15 of 
images in an image sequence, in particular, for image 
stabilization using a plurality of image regions 6, 7, 8 that 
15 each essentially cover the foreground or the background, the 
most favorable combination of information [is]may_be 
ascertained in each case from the subregions, [ in order] to 
minimize the influence of moving objects in the background. 
With the aid of a decision criterion, a moving object is 

2 0 detected in the image regions, which are, for the most part, 

covered by the background. [ ] 

If the two second image regions 6, 7 do not contain a moving 
object, and therefore, do not contain an additional movement 
25 by the moving object, which is superimposed on the image 

movement and is especially caused by a camera movement, then 
the displacement to be calculated is determined from the mean 
of the calculated regional displacements of the two first 
image regions 6, 7. If one first image region 6, 7 contains a 

3 0 ■ moving object, then displacement 15 to be determined is 

ascertained from the other image region not having a moving 
object. If both first image regions 6, 7 contain a moving 
object, then the information from second image region 8, the 
foreground subregion, is used for determining the movement. [ 
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In order to] 

To determine the regional displacement of an image region 6, 
7, 8, [the present invention uses ] two alternative methods may 
5 be used , depending on whether or not block-displacement 

information from a block-based encoding method, [e.g.] for 
example , from a block-based transmission mode (method) for 
reducing bandwidth, is accessible fbyl in a relatively simple 
[means] way . [ ] 

10 

If block-displacement information is not easily accessible 
(readily available) , then a block-matching method, by which an 
additional movement, that i[.e.]s, a local movement inside one 
of the two first image regions 6, 7, can be detected, is used 
15 for determining the regional displacement. A local movement, 
re.g.l for example, an appearing object, occurring in an image 
region 6, 7, 8 can be detected by evaluating the 
displacement -correlation values from the block-matching 
method . [ ] 

20 

To this end, the ratio of the average value of the 
displacement -correlation values to the maximum correlation 
value is compared to an adaptive threshold-value function. [In 
order t]To determine the mean of the displacement -correlation 

25 values, the sum of all of the determined correlation values is 
calculated and subsequently divided by the number of 
correlation values. The maximum displacement-correlation value 
is assumed for a particular displacement. The regional 
displacement corresponding to the maximum 

30 displacement -correlation value is assumed to be the regional 
displacement. The correlation quotient corresponds to the 
maximum displacement -correlation value divided by the mean of 
the displacement-correlation values, and is therefore 
normalized . 
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An additional movement, that i [ . e . ] s , a local movement inside 
the image region, is then detected when the correlation 
quotient is less than the value of an adaptive threshold- value 
function. The adapted threshold- value function is a function 
of the length of a displacement vector, which indicates a 
regional displacement. [In order t] To detect an additional 
movement or a local movement in one of the two first image 
regions 6, 7, the [present invention provides for the 
] correlation quotient of the first region 6, 7 in 
consideration [being] may be compared to the value of the 
adaptive threshold-value function, the value of the adaptive 
threshold-value function resulting from the vector length of 
displacement 15 for the other of the two first image regions 
6, 7. In this context, the threshold-value function [isl may be 
defined as follows: 
[ 

] - for any arbitrary length of the regional-displacement 
vector, that i [ . e . ] s , for any arbitrary displacement, 
which is less than a preselected, first threshold value, 
the value of a preselected, second threshold value is 
assumed by the threshold-value function; 
for any arbitrary length of the regional -displacement 
vector, which is greater than the preselected, first 
threshold value, the value of the preselected, second 
threshold value minus a product is assumed by the 
threshold value function, the product including a 
preselected slope parameter and a difference as factors, 
and the difference being formed from the arbitrary 
regional displacement and the preselected first threshold 
value . 



The second exemplary method for determining the displacement 
of images in an image sequence is based on the use of 
block-displacement information from a block-based encoding 
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method. As an alternative to calculating displacement, using a 
block-matching method for an image region 6, 7, 8, the 
displacement vectors of small blocks, [e.g.] for example, of 
size 8x8 or 16x16 can also be used for determining the 
regional displacement of image regions 6, 7, 8, and therefore, 
information from an encoding method can be directly used [in 
order ] to reduce the computational time. 

This approach [ i s 1 may be particularly of interest if, for 
example, hardware support allows the block-displacement 
vectors to be obtained without additional expenditure, or with 
only a small amount of additional expenditure. A local 
movement inside an image region 6, 7, 8, in particular, inside 
one of the two first image regions 6, 7, can be detected in an 
especially simple manner, when displacement vectors of small 
blocks of the image are, for example, known from an encoding 
method. 

In this context, all of the displacement vectors for blocks 
lying within one of image regions 6, 7, 8 are initially 
assigned to the corresponding image region 6, 7, 8. Separate 
frequency distributions or histograms are generated for the 
horizontal and vertical components of each of the assigned 
displacement vectors. The result for each image region 6, 7, 8 
is a first frequency distribution for the horizontal component 
and a second frequency distribution for the vertical component 
of the displacement vectors of the image blocks. 

An additional movement or a local movement is detected by 
analyzing the frequency distributions assigned to the one 
image region. A local movement is detected, if the difference 
of the positions between the principal maximum and the 
greatest secondary maximum of one of the two assigned 
frequency distributions exceeds a certain threshold value, and 
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the magnitude of the principal maxima falls below a threshold 
value . 

The determination of a local or additional movement inside an 
5 image region means that the regional displacement could not be 

calculated with sufficient reliability. Consequently, the 
calculation of the reliability factor of the 

regional -displacement determination yields a negative result 
with regard to the regional-displacement determination. 

10 

The exemplary method for determining the regional 
displacement, including its horizontal and vertical 
components, and for calculating the reliability of the 
regional -displacement determination, [can be] is described [ in 
15 a more detailed manner] as follows: 

[in order ] to calculate the horizontal component of the 
regional displacement, a first frequency distribution is 
generated for the frequencies of different values of the 
horizontal components of the block-displacement 

2 0 information, the horizontal component of the regional 

displacement corresponding to the value of the horizontal 
component of the block-displacement information, at which 
the first frequency distribution assumes its principal 
maximum; 

25 - [in order ] to calculate the vertical component of the 

regional displacement, a second frequency distribution is 
generated for the frequencies of different values of the 
vertical components of the block-displacement 
information, the vertical component of the regional 

3 0 displacement corresponding to the value of the vertical 

component of the block-displacement information, at which 
the second frequency distribution assumes its principal 
maximum; 

the reliability of the regional -displacement 
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determination is considered to be sufficient, when all of 

the following conditions are fulfilled: 

the absolute value of the difference in the position 
of the values of the horizontal component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
the first frequency distribution, is less than a 
preselected, first difference threshold; 
the absolute value of the difference in the position 
of the values of the vertical component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
the second frequency distribution, is less than a 
preselected, second difference threshold; 
the principal maximum of the first frequency 
distribution is greater than a first frequency 
threshold; and 

the principal maximum of the second frequency 
distribution is greater than a second frequency 
threshold ; 

As an alternative to the block-matching method for an image 
region 6, 7, 8 in question, block-displacement vectors of a 
block-oriented image encoding method can therefore be used to 
determine the displacement of images in an image sequence, [in 
order ] to reduce the computational time. For the 
block-matching method, a criterion was specified which is 
suited for detecting local movements inside relevant image 
region 6, 7, 8, and is synonymous to the reliability of the 
regional -displacement determination not being sufficient. A 
criterion for detecting local movements inside relevant image 
regions 6, 7, 8 was likewise specified for the 
displacement-estimation method using block-displacement 
vectors . 
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[Abstract 
Proposed is] 

ABSTRACT OF THE DISCLOSURE 
5 A device and a method for determining a displacement [ (15) of 
images in an image sequence, and a device for determining a 
displacement (15)] of images in an image sequence, which are 
specially used to compensate for a camera movement. A 
plurality of image regions [(6, 7, 8) ] of the images is 

10 available for determining the displacement [ (15)], each image 
region [ (6, 7, 8)] being provided at a preselected position of 
the images, each image region [ (6, 7, 8)] having preselected 
dimensions, in particular, a predetermined number of pixels in 
various directions of the image, and an image movement mainly 

15 caused by a camera movement being separated from an additional 
movement that is superimposed on the image movement in a 
subregion of the image to be corrected, [using the following 
steps : 

-I by calculating the probability of the movement occurring 
20 without the additional movement is calculated at different 
positions of an image [ ; 

-1 , specifying and fixing for each image region [ (6, 7, 8,)], 
the position and the dimensions [ are specified and fixed] as a 
function of the probability of the image movement occurring 
25 inside the one image region [(6, 7, 8) ]without the additional 
movement [;]_,_ and [ 

-] preferentially using at least one image region [ (6, 7)] out 
of the plurality of image regions [(6, 7, 8) is preferentially 
used ] for determining the displacement [ (15) ] . 

30 

[ (Figure 4 ) ] 

405989 
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METHOD AND DEVICE FOR DETERMINING A 
DI SPLACEMENT OF IM AGES IN AN IMAGE SEQUENCE 



Background Information 

The present invention starts out from a method for determining 
the displacement of images in an image sequence according to 
5 the species defined in the main claim, and from a device for 
determining the displacement of images in an image sequence 
according to the species defined in the other independent 
Claim 13. An image-stabilization system, which is supposed to 
substantially stabilize an image while recording with a 

10 camcorder in one's hand, is known from the article of H. 

Rindtorff, "Bildstabilisation in Consumer-Camcordern, Funktion 
und Wirkungsweise" ("Image Stabilization in Consumer 
Camcorders - Function and Mode of Operation") Fernseh- und 
Kinotechnik (Television and Cinema Technology), 49th year, no. 

15 ^, 1995. In this context, the image is split up into four 

sectors, in which movement vectors describing the displacement 
of the image are determined. A total movement vector, which 
ideally represents the displacement of the camera, is 
determined from the movement vectors of the individual 

20 sectors. 

Summary of the Invention 

In contrast, the method of the present invention, possessing 
25 the features of the main claim, has the advantage that image 
regions, which are specially utilized in the sections of an 
image in order to determine the displacement of the images, 
are distinguished by as high probability as possible, of the 
image movement occurring without the additional movement. By 
30 this means, the displacement of images of an image sequence is 
determined in a particularly reliable manner. In addition, it 
is conducive to the reliable determination of the displacement 



of images in an image sequence, when the position and the 
dimensions of the image regions are fixed. This especially 
allows the displacement to be reliably determined for a 
special scenario of moving picture communication (full -motion 
5 video), using relatively little processing time. A first image 
region, which is preferentially utilized from the plurality of 
image regions to determine the displacement, may be used alone 
for determining the displacement, by means of selecting its 
position and its dimensions inside the images, so that other 
10 image regions do not have to be considered, and thus, the 

processing time for determining the displacement of images is 
decreased. 

Advantageous further refinements and improvements of the 
15 method indicated in the main claim are rendered possible by 
measures specified in the dependent claims. 

It is particularly advantageous that a first image of the 
image sequence is corrected by determining the displacement, 
2 0 either from image data of the first image and image data of a 
second image of the image sequence that preferably precedes 
the first image directly, or from image data of the first 
image and image data of a second input image of the image 
sequence that preferably precedes the first image directly; a 

2 5 regional displacement of an image region being utilized as a 

displacement, and the regional displacement of an image region 
of the plurality of the image regions either being determined 
from the image data of the first image and the second image 
inside the one image region, or from the image data of the 

3 0 first image and the second input image inside the one image 

region. In this manner, only the image data of the first and 
the second images inside the image region, which is used for 
determining the displacement of the image, are needed for 
processing, which means that the processing time is reduced. 

35 

In addition, it is advantageous that the position and the 
dimensions of the at least one first image region are selected 

2 



in such a manner, that the at least one first image region of 
the images to be corrected is mainly filled in by the image 
background. Since, as a rule, the image background, rarely 
exhibits additional movements from one image to another image, 
5 between which the displacement should be determined, in which 
case the additional movements are superimposed on the image 
movement that is especially caused by the unintentional 
movement of a camera, a first image region of the images to be 
corrected is advantageously used for determining the 
10 displacement, when it is filled in by the image background. 

Furthermore, it is advantageous that the position and the 
dimensions of at least one second image region are selected in 
such a manner, that the at least one second image region of 

15 the images to be corrected is mainly filled in by the image 

foreground. This allows the displacement of images of an image 
sequence to be determined in an especially simple manner, and 
with high accuracy, when, e.g. the image background is 
subjected to a sharp, additional movement, which is 

2 0 superimposed on the image movement that may be caused by the 
unintentional camera movement. 



In order to determine the displacement, it is also 
advantageous that the at least one first and/or the at least 

25 one second image region is used as a function of a margin of 

reliability (reliability factor) , for determining the regional 
displacement. This allows the determination of the region 
displacement to be checked, because a measure for the 
reliability of the regional -displacement determination is 

30 available inside the method. Therefore, the determination of a 
reliability factor for the regional -displacement determination 
is primarily used to simply and reliably separate the image 
movement from the additional movement superimposed on it. 



Furthermore, it is advantageous that two first image regions 
and an individual second image region are available for image 
correction, the displacement being determined in one of the 



following three ways, in the order of descending priority: 

from the mean of the regional displacements of the two 
first image regions, when the reliability factor of the 
5 regional -displacement determination of the two first 

image sequences is considered to be sufficient; 
from the regional displacement of one of the two first 
image regions, in which the reliability of the 
regional -displacement determination is considered to be 
10 sufficient; 

from the regional displacement of the second image 
region . 

This allows different image regions of different priority to 

15 be used for determining the displacement of images in an image 
sequence. In particular, it is possible to preferentially 
utilize image regions, which are mainly filled in by the image 
background, for determining the displacement of images; the 
use of the mean of the regional displacements of the two first 

20 image regions resulting in a reliable determination of the 

displacement. The second priority for determining the 
displacement, using the regional displacement calculated from 
the two first image regions, is especially selected in order 
to minimize the influence of objects moving in the background, 

25 since an object moving into one of the two first image regions 

leads to the regional displacement of the other of the two 
image regions being used for determining the displacement of 
images in an image sequence. On the third level of priority, 
the displacement of images from the regional displacement of 

30 the second image region, especially from the regional 

displacement of an image region that is mainly filled in by 
the image foreground, is used. 

In addition, it is advantageous that the method is used for a 
35 head- shoulder recording situation, the two first image regions 
being selected to be in a lateral image region, to the left 
and right of, and preferably symmetrical to, the vertical 

4 



center line of a predetermined rectangular image, the distance 
of the two first image regions from the bottom image margin 
being selected to be greater than the distance of the two 
first image regions from the top image margin, the second 
5 image region being selected to be near the center of the 

image, preferably symmetric to the vertical center line of the 
rectangular image, and the distance of the second image region 
from the top image margin being preferably selected to be 
greater than the distance of the second image region from the 

10 bottom image margin. In the case of a head- shoulder recording 
situation, such a choice of image regions particularly allows 
the information from the image regions to be sensibly used for 
determining the regional displacements, and sensibly used to 
determine the displacement of images in an image sequence, by 

15 prioritizing the utilization of regional displacements. 

In addition, it is advantageous that, for two image regions, 
preferably the two first image regions, the regional 
displacement and the reliability of the regional -displacement 

2 0 determination are determined by a method having the following 

steps : 

the regional displacement and a correlation quotient are 
calculated for each of the two image regions ; 
a threshold value function is determined as a function of 
25 each ascertained regional displacement of the two image 

regions ; 

each correlation quotient of one of the two image regions 
is compared to the value, which results from a threshold 
value function for the regional displacement of the other 

3 0 of the two image regions ; 

the reliability of the regional -displacement 
determination for each of the two image regions is viewed 
as being sufficient, when the correlation quotient 
corresponding to the one image region is greater than the 
35 value of the threshold- value function to be compared to 

the correlation quotient . 
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In this manner, the reliability factors for the regional 
displacement of the two first image regions are not determined 
independently of each other, but rather the ascertained 
regional displacement of one of the two first image regions 
5 determines the value, which the threshold- value function 

assumes, and to which the regional displacement of the other 
of the two first image regions is compared. Therefore, the 
regional displacement of one image region and the reliability 
factor of the regional displacement are not determined 
10 independently of movements, especially regional displacements, 

in other parts of an image. 

Furthermore, it is advantageous that the correlation quotient 
of one of the two image regions is determined according to a 
15 method having the following steps: 

using a block-matching method, displacement-correlation 
values are determined for several possible regional 
displacements; 

the regional displacement, for which the 
20 displacement-correlation values are maximized, is 

regarded as the regional displacement of the image 
region; 

the correlation quotient is formed by dividing the 
maximum displacement-correlation value by the average 
25 value of the calculated displacement -correlation values. 



In this manner, the regional displacement is first of all 
determined using the entire image region, since the 
displacement -correlation values are calculated by forming the 
3 0 sum over (of) all the image points of the image region in 
consideration. Secondly, the correlation quotient is 
normalized, since the maximum displacement -correlation value, 
from which the regional displacement results, is divided by 
the mean of the calculated displacement-correlation values. 

35 

In addition, it is advantageous that, for each arbitrary 
regional displacement less than a preselected first threshold 

6 



value, the value of a preselected second threshold value is 
assumed by the threshold-value function, and that, for each 
arbitrary regional displacement greater than the preselected 
first threshold value, the value of the preselected second 
5 threshold value minus a product is assumed by the 

threshold-value function; the product including a preselected 
slope parameter and a difference as factors, and the 
difference being formed from the arbitrary regional 
displacement and the preselected first threshold value. The 

10 advantage of this is that, for larger, ascertained regional 

displacements of an image region, the regional displacement is 
allowed a sufficient margin of reliability, even when the 
correlation coefficient is smaller, which, in the case of a 
small, ascertained regional displacement, would indicate a 

15 correlation that is not as good. Due to the dependence of the 
two first image regions in the calculation of the reliability 
factor for the determination of the regional displacement, the 
result of the structure of the threshold-value function is 
that, in response to a relatively large regional displacement 

2 0 of one of the two first image regions, the required 

reliability of the regional -displacement determination, i.e. 
the required magnitude of the correlation coefficient, are not 
as high. 

25 In a further, specific embodiment of the method for 

determining a displacement of images in an image sequence, it 
is advantageous that block-displacement information from a 
block-based encoding method is used for determining the 
regional displacement; the block-displacement information, 

3 0 preferably displacement vectors, of blocks lying inside the 

specific image region being considered for each of the image 
regions. This results in an especially simple and 
cost-effective implementation of the method according to the 
present invention, especially when, e.g. block-displacement 
35 information, in particular block-displacement vectors, of 

small image blocks can be obtained without additional 
expenditure, or with only a small amount of additional 
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expenditure . 

In addition, it is particularly advantageous for the second 
specific embodiment of the method according to the present 
invention, that a method having the following steps is 
implemented for each of the image regions, in order to 
determine the regional displacement that has a horizontal and 
vertical component, and in order to calculate the reliability 
factor of the regional -displacement determination. 

In order to calculate the horizontal component of the 
regional displacement, a first frequency distribution is 
generated for the frequencies of different values of the 
horizontal components of the block-displacement 
information, the horizontal component of the regional 
displacement corresponding to the value of the horizontal 
component of the block-displacement information, at which 
the first frequency distribution assumes its principal 
(global, absolute) maximum; 

in order to calculate the vertical component of the 
regional displacement, a second frequency distribution is 
generated for the frequencies of different values of the 
vertical components of the block-displacement 
information, the vertical component of the regional 
displacement corresponding to the value of the vertical 
component of the block-displacement information, at which 
the second frequency distribution assumes its principal 
maximum; 

the reliability of the regional-displacement 
determination is considered to be sufficient, when all of 
the following conditions are fulfilled; 

the absolute value of the difference in the position 
of the values of the horizontal component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary (local, 
relative) maximum of the first frequency 
distribution, is less than a preselected, first 
difference threshold; 



the absolute value of the difference in the position 
of the values of the vertical component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
5 the second frequency distribution, is less than a 

preselected, second difference threshold; 
the principal maximum of the first frequency 
distribution is greater than a first frequency 
threshold; and 

10 - the principal maximum of the second frequency 

distribution is greater than a second frequency 
threshold. 

Using simple means, in particular, by considering the 
15 frequencies of the occurring, horizontal and vertical 
components of the already present block-displacement 
information, it is thereby possible to, first of all, 
determine the regional displacement of an image region, and 
secondly, to calculate the reliability factor of the 
20 regional -displacement determination. 

The device of the present invention, which is for determining 
a displacement of images in an image sequence, and possesses 
the features of the other independent Claim 13, has the 

25 advantage that the displacement can especially be determined 
in a quicker and simpler manner by implementing the method 
steps of the method according to the present invention, in a 
microchip or on a printed circuit board, which results in the 
method of the present invention being carried out in a quicker 

3 0 and more cost-effective manner. 

Brief Description of the Drawing 

An exemplary embodiment of the present invention is 
35 represented in the drawing and explained in detail in the 
following description. 
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The figures show: 



a block diagram of the device according to the 
present invention; 

the principal of image stabilization, by determining 
a displacement of images in an image sequence; 

a flowchart for image stabilization; and 

an example of selecting image regions inside an 
image, in order to implement the method according to 
the present invention. 

15 Description of the Exemplary Embodiment 

Figure 1 shows a block diagram of the device according to the 
present invention, for determining a displacement of images in 
an image sequence. The device according to the present 

2 0 invention includes an input 10, an output 20, a 

displacement-detection circuit 100, and a magnification 
circuit 200. Displacement -detection circuit 100 includes a 
regional -displacement detector 110, an image- storage device 
(display store) 120, and a microcomputer 130. In addition, 
25 displacement-detection circuit 100 includes an input, which is 
not shown and is connected to input 10 of the device according 
to the present invention, to regional -displacement detector 
110, and to image-storage device 120. In addition, 
displacement-detection circuit 100 includes an output, which 

3 0 is not shown and is connected to microcomputer 13 0; 

regional -displacement detector 110 also being connected to 
microcomputer 130. Magnification circuit 200 includes two 
inputs not shown and an output, which is connected to output 
20 of the device. The two inputs of magnification circuit 200 
35 are each connected to a magnification processor 210, one of 

the two inputs of magnification circuit 200 being connected to 
input 10 of the device, and the other of the two inputs of 

10 



Figure 1 



Figure 2 



Figure 3 
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Figure 4 



magnification circuit 200 being connected to the output of 
displacement -detection circuit 100. 

An example of image stabilization, e.g. compensating for 
5 camera movements, is represented in Figure 2. In this context, 
a first input image 13 is corrected to form a first output 
image 23, use being made of image information regarding a 
second input image 11, which preferably precedes the first 
input image in time. 

10 

Second input image 11 includes an image detail, which is 
enlarged by magnification to form second output image 21. In 
this context, second input image 11 is completely recorded by 
the camera, but a user of the device according to the present 
15 invention may only see the image detail in magnified form, as 

second output image 21. In this case, the image detail is 
referred to as second image 12, or also as corrected second 
image 12 . 

2 0 According to a first specific embodiment of the method of the 
present invention, corrected second image 12 is used to 
correct first input image 13 . First input image 13 also 
includes an image detail, which is referred to here as 
uncorrected first image 14. A displacement 15 can be 

2 5 determined by comparing uncorrected first image 14 to second 
image 12, i.e. to corrected second image 12, so that 
uncorrected first image 14 can be converted into a corrected 
first image 16, using displacement 15. In particular, not all 
of the image data are utilized in comparing uncorrected first 

30 image 14 to second image 12, but rather, just the image data 
from image regions (not shown) of first image 14 and second 
image 12 . 

According to a second specific embodiment of the method of the 
35 present invention, second image 11 is used to correct first 

input image 13. Displacement 15 can be determined by comparing 
first input image 13 to second input image 11, so that 

11 



uncorrected first image 14 can be converted into corrected 
first image 16, using displacement 15. In particular, not all 
of the image data are utilized in comparing first input image 
13 to second input image 11, but rather, just the image data 
5 from image regions (not shown) of first input image 13 and 
second input image 11. ■ 

Corrected first image 16 can now be shown to the user as a 
first output image 23, in accordance with both the first 
10 specific embodiment and the second specific embodiment of the 
method according to the present invention. Compared to the 
second output image, the displacement of first input image 13 
or corrected first image 16 can no longer be observed in first 
output image 2 3 . 

15 

The essential method steps of the method according to the 
present invention are represented in Figure 3 by a flow chart, 
using the correction of first input image 13 as an example. 
The regional displacements of image regions are determined in 

20 a first sequence step 30. In this connection, first input 

image 13 is compared to corrected second image 12, which 
especially precedes it in time, or the first input image is 
compared to second input image 11, which especially precedes 
it in time. In sequence step 40, the displacement of first 

2 5 image 14 is subsequently determined in accordance with the 
method of the present invention. In third sequence step 50, 
uncorrected first image 14 is displaced by displacement vector 
15. The result of this operation is corrected first image 16. 
Subsequently, corrected first image 16 is enlarged in fourth 

30 sequence step 60, which results in first output image 23. 

In order to correct a further input image in accordance with 
the method of the present invention, the result of third 
sequence step 50 can be made available to first sequence step 
35 30 by storing it, preferably in image -storage device 120. The 
result of third sequence step 50 is corrected first image 16, 
which therefore appears in the place of corrected second image 



12, and, in connection with the correction of the further 
input image, is used for determining regional displacements in 
first sequence step 30. 



5 As an alternative, first input image 13 or uncorrected first 
image 14 can, together with calculated displacement 15, also 
be saved in image- storage device 12 0, in order to correct an 
additional input image . 

10 Represented in Figure 4 is an example of a distribution of 
image details 6, 7, 8 inside first input image 14. In this 
context, two first image regions 6, 7 are selected to be 
symmetric to the vertical center line of given, rectangular 
first image 14, especially for using the method of the present 

15 invention in a head- shoulder recording situation. In this 

context, the distance of the two first image regions 6, 7 from 
the bottom image margin is greater than the distance of the 
two first image regions 6, 7 from the top image margin. 
A second image region 8 is selected to be near the center of 

20 first image 14, preferably symmetric to the vertical center 

line of the rectangular image, the distance of second image 
region 8 from the top image margin being preferably selected 
to be greater than the distance of the second image region 
from the bottom image margin. 

25 

The method of the present invention for determining a 
displacement 15 of images in an image sequence can preferably 
be used for image stabilization, in order to compensate for 
camera movements in consumer video cameras (camcorders) . The 

30 digital, full -motion image stabilization of the method renders 
it particularly suitable for mobile moving-picture 
communication (mobile full -motion video) devices. Movements 
caused by the mobile use of a full -motion video device should 
be reduced or eliminated to the greatest possible extent. The 

3 5 basic principle of the method is to derive (deduce) the camera 
movement from the relative displacement of consecutive images, 
and to extract the subregion, e.g. corrected first image 16, 



from an input image, e.g. first input image 13, the subregion 
compensating for the camera movement, based on the ascertained 
displacement 15, in particular of corrected first image 16, 
with respect to uncorrected first image 14 . 

5 

In the method according to the present invention, a plurality 
of image regions 6, 7, 8 are made available for determining 
displacement 15. The image regions can be clearly (uniquely) 
determined inside the image, using their positions and their 

10 dimensions. By advantageously selecting the position and the 

dimensions of image regions 6, 7, 8, an image movement that is 
mainly generated by a camera movement can now be separated in 
a particularly simple manner, from an additional movement, 
which is superimposed on the image movement in subregions of 

15 the image to be corrected. In addition, the probability of the 
image movement occurring at various positions of the images of 
an image sequence without the additional movement, is 
determined. The result of this is preferential portions 
(sections) inside the image, which can be used for separating 

20 the image movement from the additional movement. For example, 
the selection of image regions 6, 7, 8 represented in Figure 4 
is especially advantageous for the special recording situation 
of a speaker who is in the center of the image. The special 
characteristics of the recording situation are used as a 

25 priori knowledge in selecting and fixing image regions 6, 7, 

8. In this recording situation, it can be assumed that the two 
first image regions 6, 7 are essentially in the background of 
the image and second image region 8 is essentially in the 
foreground. This means that the two first image regions 6, 7 

3 0 are mainly filled up with image data from the image 

background, and that second image region 8 is mainly filled up 
with image data from the image foreground. This allows the two 
first image regions 6, 7 to be advantageously prioritized, so 
that the determination of a displacement 15 of images in an 

35 image sequence is preferably derived from the determination of 

the regional displacement of the two first image regions 6, 7. 
Second image region 8 is only used for determining 
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displacement 15 of images, when the use of regional 
displacements from the two first image regions 6, 7 merely 
permits a regional displacement, which is encumbered by a 
large lack of reliability or an insufficiently large 
5 reliability factor. In the specified example, this 

prioritization is synonymous to the image stabilization being 
preferentially carried out using image -background information. 
However, this method does not exclusively relate to the 
differentiation and variable prioritization of background and 
10 foreground information, in order to determine a displacement 

of images in an image sequence, but also relates, for example, 
to the use of criteria such as edge detection, absence of edge 
detection, or similar. 

15 In the specified example in QCIF format {Quarter Common 
Intermediate Format) , the two first image regions 6, 7 
typically extend 120 pixels in the vertical direction and 40 
pixels in the horizontal direction. In this image format, 
second image region 8 typically extends over 135 pixels in the 

2 0 vertical direction and 85 pixels in the horizontal direction. 

In the method for determining a displacement 15 of images in 
an image sequence, in particular, for image stabilization 
using a plurality of image regions 6, 7, 8 that each 

2 5 essentially cover the foreground or the background, the most 

favorable combination of information is ascertained in each 
case from the subregions, in order to minimize the influence 
of moving objects in the background. With the aid of a 
decision criterion, a moving object is detected in the image 

3 0 regions, which are, for the most part, covered by the 

background. If the two second image regions 6, 7 do not 
contain a moving object, and therefore, do not contain an 
additional movement by the moving object, which is 
superimposed on the image movement and is especially caused by 
35 a camera movement, then the displacement to be calculated is 
determined from the mean of the calculated regional 
displacements of the two first image regions 6, 7. If one 



first image region G, 7 contains a moving object, then 
displacement 15 to be determined is ascertained from the other 
image region not having a moving object. If both first image 
regions 6, 7 contain a moving object, then the information 
5 from second image region 8, the foreground subregion, is used 

for determining the movement . In order to determine the 
regional displacement of an image region 6, 7, 8, the present 
invention uses two alternative methods, depending on whether 
or not block-displacement information from a block-based 

10 encoding method, e.g. from a block-based transmission mode 
(method) for reducing bandwidth, is accessible by simple 
means. If block-displacement information is not easily- 
accessible (readily available) , then a block-matching method, 
by which an additional movement, i.e. a local movement inside 

15 one of the two first image regions 6, 7, can be detected, is 
used for determining the regional displacement. A local 
movement, e.g. an appearing object, occurring in an image 
region 6, 7, 8 can be detected by evaluating the 
displacement -correlation values from the block-matching 

2 0 method. To this end, the ratio of the average value of the 
displacement -correlation values to the maximum correlation 
value is compared to an adaptive threshold-value function. In 
order to determine the mean of the displacement-correlation 
values, the sum of all of the determined correlation values is 

2 5 calculated and subsequently divided by the number of 

correlation values. The maximum displacement -correlation value 
is assumed for a particular displacement. The regional 
displacement corresponding to the maximum 

displacement-correlation value is assumed to be the regional 
30 displacement. The correlation quotient corresponds to the 

maximum displacement -correlation value divided by the mean of 
the displacement-correlation values, and is therefore 
normal i zed . 

35 An additional movement, i.e. a local movement inside the image 
region, is then detected when the correlation quotient is less 
than the value of an adaptive threshold-value function. The 

16 



adapted threshold-value function is a function of the length 
of a displacement vector, which indicates a regional 
displacement. In order to detect an additional movement or a 
local movement in one of the two first image regions 6, 7, the 
5 present invention provides for the correlation quotient of the 
first region 6, 7 in consideration being compared to the value 
of the adaptive threshold- value function, the value of the 
adaptive threshold-value function resulting from the vector 
length of displacement 15 for the other of the two first image 
10 regions 6, 7. In this context, the threshold-value function is 
defined as follows: 

for any arbitrary length of the regional -displacement 
vector, i.e. for any arbitrary displacement, which is 
15 less than a preselected, first threshold value, the value 

of a preselected, second threshold value is assumed by 
the threshold-value function; 

for any arbitrary length of. the regional -displacement 
vector, which is greater than the preselected, first 

2 0 threshold value, the value of the preselected, second 

threshold value minus a product is assumed by the 
threshold value function, the product including a 
preselected slope parameter and a difference as factors, 
and the difference being formed from the arbitrary 
25 regional displacement and the preselected first threshold 

value . 

The second method for determining the displacement of images 
in an image sequence is based on the use of block-displacement 

3 0 information from a block-based encoding method. As an 

alternative to calculating displacement, using a 
block-matching method for an image region 6, 7, 8, the 
displacement vectors of small blocks, e.g. of size 8x8 or 
16x16 can also be used for determining the regional 
3 5 displacement of image regions 6, 7, 8, and therefore, 

information from an encoding method can be directly used in 
order to reduce the computational time. This approach is 



particularly of interest if, for example, hardware support 
allows the block-displacement vectors to be obtained without 
additional expenditure, or with only a small amount of 
additional expenditure. A local movement inside an image 
5 region 6, 7, 8, in particular, inside one of the two first 
image regions 6, 7, can be detected in an especially simple 
manner, when displacement vectors of small blocks of the image 
are, for example, known from an encoding method. In this 
context, all of the displacement vectors for blocks lying 

10 within one of image regions 6, 7, 8 are initially assigned to 

the corresponding image region 6 , 7 , 8 . Separate frequency 
distributions or histograms are generated for the horizontal 
and vertical components of each of the assigned displacement 
vectors. The result for each image region 6, 7, 8 is a first 

15 frequency distribution for the horizontal component and a 

second frequency distribution for the vertical component of 
the displacement vectors of the image blocks. An additional 
movement or a local movement is detected by analyzing the 
frequency distributions assigned to the one image region. A 

2 0 local movement is detected, if the difference of the positions 

between the principal maximum and the greatest secondary 
maximum of one of the two assigned frequency distributions 
exceeds a certain threshold value, and the magnitude of the 
principal maxima falls below a threshold value. 

25 

The determination of a local or additional movement inside an 
image region means that the regional displacement could not be 
calculated with sufficient reliability. Consequently, the 
calculation of the reliability factor of the 

3 0 regional -displacement determination yields a negative result 

with regard to the regional -displacement determination. 

The method for determining the regional displacement, 
including its horizontal and vertical components, and for 
35 calculating the reliability of the regional -displacement 

determination, can be described in a more detailed manner as 
follows : 
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in order to calculate the horizontal component of the 
regional displacement, a first frequency distribution is 
generated for the frequencies of different values of the 
horizontal components of the block-displacement 
information, the horizontal component of the regional 
displacement corresponding to the value of the horizontal 
component of the block-displacement information, at which 
the first frequency distribution assumes its principal 
maximum ; 

in order to calculate the vertical component of the 
regional displacement, a second frequency distribution is 
generated for the frequencies of different values of the 
vertical components of the block-displacement 
information, the vertical component of the regional 
displacement corresponding to the value of the vertical 
component of the block-displacement information, at which 
the second frequency distribution assumes its principal 
maximum ; 

the reliability of the regional -displacement 
determination is considered to be sufficient, when all of 
the following conditions are fulfilled: 

the absolute value of the difference in the' position 
of the values of the horizontal component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
the first frequency distribution, is less than a 
preselected, first difference threshold; 
the absolute value of the difference in the position 
of the values of the vertical component of the 
block-displacement information, which correspond to 
the principal maximum and the secondary maximum of 
the second frequency distribution, is less than a 
preselected, second difference threshold; 
the principal maximum of the first frequency 
distribution is greater than a first frequency 
threshold; and 

the principal maximum of the second frequency 



distribution is greater than a second frequency- 
threshold; 



As an alternative to the block-matching method for an image 
5 region 6, 7, 8 in question, block-displacement vectors of a 

block-oriented image encoding method can therefore be used to 
determine the displacement of images in an image sequence, in 
order to reduce the computational time. For the block-matching 
method, a criterion was specified which is suited for 

10 detecting local movements inside relevant image region 6, 7, 
8, and is synonymous to the reliability of the 
regional -displacement determination not being sufficient. A 
criterion for detecting local movements inside relevant image 
regions 6, 7, 8 was likewise specified for the 

15 displacement-estimation method using block-displacement 
vectors . 
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What is claimed is: 



1. A method for determining a displacement (15) of images in 
an image sequence, in particular for compensating for a camera 
movement, a plurality of image regions (6, 7, 8) of the images 
being available for determining the displacement (15) , each 
image region (6, 7, 8) being provided at a preselected 
position of the images, and each image region (6, 7, 8) having 
preselected dimensions, in particular, predetermined numbers 
of pixels in various directions of the image, wherein an image 
movement mainly caused by a camera movement is separated from 
an additional movement, which is superimposed on the image 
movement in subregions of the image to be corrected; using the 
following steps: 

the probability of the image movement occurring 
without the additional movement is calculated at 
different positions of an image; 

for each image region (6, 7, 8,), the position and 
the dimensions are specified and fixed as a function 
of the probability of the image movement occurring 
inside the one image region (6, 7, 8) without the 
additional movement; and 

at least one image region (6, 7) out of the 
plurality of image regions (6, 7, 8) is 
preferentially used for determining the displacement 
(15) . 

2. The method as recited in Claim 1, wherein a first image 
(14) of the image sequence is corrected by determining the 
displacement (15) , either from image data of the first 
image -sequence image (14) and image data of a second 
image- sequence image (12) that preferably precedes the first 
image directly, or from image data of the first image- sequence 
image (14) and image data of a second input (starting) 
image -sequence image (11) that preferably precedes the first 
image directly, a regional displacement of an image region (6, 
7, 8) being utilized as a displacement (15) , the regional 
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displacement of an image region (6, 7, 8) of the plurality of 
the image regions (6, 7, 8) either being determined from the 
image data of the first image (14) and the second image (12) 
inside the one image region (6, 7, 8) , or from the image data 
of the first image (14) and the second starting image (11) 
inside the one image region (6, 7, 8) . 

3. The method as recited in Claim 1 or 2 , wherein the position 
and the dimensions of the at least one first image region (6, 
7) are selected in such a manner, that the at least one first 
image region (6, 7) of the images to be corrected is mainly 
filled in by the image background. 

4. The method as recited in one of the preceding claims, 
wherein the position and the dimensions of the at least one 
second image region (8) are selected in such a manner, that 
the at least one second image region (8) of the images to be 
corrected is mainly filled in by the image foreground. 

5. The method as recited in one of the preceding claims, 
wherein, in order to determine the displacement (15) , the at 
least one first and/or the at least one second image region 
(6, 7, 8) is used as a function of a margin of reliability 
(reliability factor) for determining the regional 
displacement . 

6. The method as recited in one of the preceding claims, 
wherein two first image regions (6, 7) and an individual 
second image region (8) are available for image correction, 
the displacement (15) being determined in one of the following 
three ways, in the order of descending priority: 

from the mean of the regional displacements of the 
two first image regions (6, 7), when the reliability 
of the regional-displacement determinations for the 
two first image regions (6, 7) is considered to be 
sufficient; 

from the regional displacement of one of the two 
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first image regions (6, 7) , in which the reliability 
of the regional -displacement determination is 
considered to be sufficient; 

from the regional displacement of the second image 
region (8) . 

7. The method as recited in Claim 6, wherein the method is 
used for a head-shoulder recording situation, the two first 
image regions (6,7) being selected to be in a lateral region, 
to the left and right of, and preferably symmetrical to the 
vertical center line of a predetermined rectangular image, the 
distance of the two first image regions (6, 7) from the bottom 
image margin being selected to be greater than the distance of 
the two first image regions (6, 7) from the top image margin, 
the second image region (8) being selected to be near the 
center of the image, preferably symmetric to the vertical 
center line of the rectangular image, and the distance of the 
second image region (8) from the top image margin being 
preferably selected to be greater than the distance of the 
second image region (8) from the bottom image margin. 

8. The method as recited in one of Claims 5 through 7, 
wherein, for two image regions (6, 7) , preferably the two 
first image regions (6, 7), the regional displacement and the 
reliability of the regional -displacement determination are 
determined by a method having the following steps: 

the regional displacement and a correlation quotient 
are calculated for each of the two image regions (6, 
7) ; 

a threshold value function is determined as a 
function of each ascertained regional displacement 
of the two image regions (6, 7) ; 

each correlation quotient of one of the two image 
regions (6, 7) is compared to the value, which 
results from a threshold value function for the 
regional displacement of the other of the two image 
regions (6, 7); and 
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the reliability factor of the regional -displacement 
determination of each of the two image regions (6, 
7) is viewed as being sufficient, when the 
correlation quotient corresponding to the one image 
region (6, 7) is greater than the value of the 
threshold value function to be compared to the 
correlation quotient. 

9. The method as recited in Claim 8, wherein the correlation 
quotient of one of the two image regions (6, 7) is determined 
according to a method having the following steps: 
using a block-matching method, 

displacement -correlation values are determined for 
several possible regional displacements; 
the regional displacement, for which the 
displacement-correlation values are maximized, is 
regarded as the regional displacement of the image 
region (6, 7) ; and 

the correlation quotient is calculated by dividing 
the maximum displacement-correlation value by the 
mean of the determined displacement-correlation 
values . 

10. The method as recited Claim 8, wherein, for each arbitrary 
regional displacement that is less than a preselected first 
threshold value, the value of a preselected second threshold 
value is assumed by the threshold-value function, and, for 
each arbitrary regional displacement that is greater than the 
preselected first threshold value, the value of the 
preselected second threshold value minus a product is assumed 
by the threshold- value function; the product including a 
preselected slope parameter and a difference as factors, and 
the difference being formed from the arbitrary regional 
displacement and the preselected first threshold value. 

11. The method as recited in Claims 5 through 7, wherein 
block-displacement information from a block-based encoding 
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method is used for determining the regional displacement; for 
each of the image regions (6, 7, 8) , the block-displacement 
information, preferably displacement vectors, of blocks lying 
inside the specific image region (6, 7, 8) being considered. 

12. The method as recited in Claim 11, wherein a method having 
the following steps is implemented for each of the image 
regions, in order to determine the regional displacement, 
including a horizontal and a vertical component, and in order 
to calculate the reliability of the regional -displacement 
determination : 

in order to calculate the horizontal component of 
the regional displacement, a first frequency 
distribution is generated for the frequencies of 
different values of the horizontal component of the 
block-displacement information, the horizontal 
component of the regional displacement corresponding 
to the value of the horizontal component of the 
block-displacement information, at which the first 
frequency distribution assumes its principal 
maximum; 

in order to calculate the vertical component of the 
regional displacement, a second frequency 
distribution is generated for the frequencies of 
different values of the vertical component of the 
block-displacement information, the vertical 
component of the regional displacement corresponding 
to the value of the vertical component of the 
block-displacement information, at which the second 
frequency distribution assumes its principal 
maximum; 

the reliability of the regional -displacement 
determination is considered to be sufficient, when 
all of the following conditions are fulfilled: 

the absolute value of the difference in the 
position of the values of the horizontal 
component of the block-displacement 
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information, which correspond to the principal 
maximum and the secondary maximum of the first 
frequency distribution, is less than a 
preselected, first difference threshold; 
the absolute value of the difference in the 
position of the values of the vertical 
component of the block-displacement 
information, which correspond to the principal 
maximum and the secondary maximum of the second 
frequency distribution, is less than a 
preselected second difference threshold; 
the principal maximum of the first frequency 
distribution is greater than a first frequency 
threshold; and 

the principal maximum of the second frequency 
distribution is greater than a second frequency 
threshold . 

13. A device for determining a displacement (15) of images in 
an image sequence, according to a method as recited in one of 
the preceding claims, wherein the device includes a 
displacement -detection circuit (100) , the 
displacement-detection circuit (100) determining the 
displacement (15) . 



26 



Abstract 

Proposed is a method for determining a displacement (15) of 
images in an image sequence, and a device for determining a 
5 displacement (15) of images in an image sequence, which are 
specially used to compensate for a camera movement. A 
plurality of image regions (6, 7, 8) of the images is 
available for determining the displacement (15), each image 
region (6, 7, 8) being provided at a preselected position of 

10 the images, each image region (6, 7, 8) having preselected 

dimensions, in particular, a predetermined number of pixels in 
various directions of the image, and an image movement mainly 
caused by a camera movement being separated from an additional 
movement that is superimposed on the image movement in a 

15 subregion of the image to be corrected, using the following 
steps : 

- the probability of the movement occurring without the 
additional movement is calculated at different positions of an 
image ; 

2 0 - for each image region (6, 7, 8,), the position and the 

dimensions are specified and fixed as a function of the 
probability of the image movement occurring inside the one 
image region (6, 7, 8) without the additional movement; and 

- at least one image region (6, 7) out of the plurality of 

2 5 image regions (6, 7, 8) is preferentially used for determining 
the displacement (15) . 

(Figure 4) 

405865 
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